Greenfly SAU, LLC

The Spectrum Collaboration Challenge (a.k.a. “SC2”) was a project by the Information Innovation office at DARPA. To kick-start AI development for radio spectrum sharing, Johns Hopkins APL constructed a radio testbed containing 128 radio emulation nodes. This included 21 server racks in a dedicated APL data center, including the radio nodes and support equipment.

Greenfly was brought on board to provide Devops expertise. Devops challenges included gaps in automation and monitoring. We also provided Ops support for moving the data center to the final competition in Los Angeles at Mobile World Congress 2019.

Technical Challenges

During user radio experiments, users reported persistent variability of system clocks. The radio nodes used NTP to synchronize time from a master Microsemi clock source. We first gathered additional metrics by using Puppet to install and configure the Splunk forwarder. This set up each radio node to forward detailed NTP metrics to the project’s Splunk server.

Greenfly constructed new dashboards to visualize and analyze the NTP metrics. These in turn gave us the insight that radio node clocks were adversely affected by temperature changes. For instance, if the temperature increased 5 degrees Fahrenheit, the radio node’s clock would speed up. When that happened, the NTP daemon required several hours to completely resynchronize the radio node’s clock.

Results

Finding the root cause of the variability and its extent allowed the project greater confidence in setting bounds on acceptable clock consistency. We were also able to focus on temperature stability, and thus reduce clock variability.

Transition

Greenfly helped transition this project to NSF PAWR program ownership.