Greenfly SAU, LLC

The Plan X project was a study of next-generation cyber operations by the Information Innovation office at DARPA. During the project there were between five and ten teams supporting diverse mission areas. Greenfly led the deployment effort. The challenge for the deployment team was to create a unified software build based on all of the latest code of every team.

Technical Challenges

Deployment started out using Jenkins as its testing and build tool, but it quickly became clear that Jenkins was unable to guarantee consistency during builds between the project’s 35 software repositories. This is called the dreaded diamond problem. To address this problem, deployment switched to GoCD.

Switching to GoCD introduced a new problem: builds frequently became stuck. Because GoCD enforces consistency between all of its build components, any build failure causes “siblings” to ignore the newest/failing build source. In the image below, when C fails, B also won’t build!

Once this problem was identified, Greenfly wrote tools to identify both the blocking component, and the source commit causing the blocker. This enabled the teams to identify problem spots, and resolve the blockers.

Results

The deployment team enabled Plan X to release 30 builds per day. The software was automatically installed on test systems, which meant the latest code was available for demos and manual software testing.

Transition

Greenfly helped transition this project to DoD/Army ownership. This included handing off usage and maintenance of the automated build system.