Approachable Release Automation
By: Jordan Place, Software Engineer
Measures for Justice (MFJ) develops tools that help communities reshape how the criminal justice system works. Within MFJ, the software engineering team builds the frontend of our data platform, Commons, and our Data Portal. We’re constantly refining how data can be explored and working to bring new data online.
A core part of our workflow is releasing new code to measuresforjustice.org, so people can use what we’ve built. We recently realized that only a small subset of our team participates in releasing though. Ideally, we’d have wide participation so that there’s redundancy and innovation in this crucial team function. To encourage participation, we set out to make our release automation more approachable.
The former automation ran on Jenkins infrastructure that is shared across the organization. This infrastructure experienced regular failures that disrupted our build and deploy jobs. Our team neither had the expertise nor mandate to address these infrastructure failures, so we manually restarted our jobs whenever this happened. De facto, just a couple vigilant developers became responsible for this work.
The jobs themselves were handed down to us by a past generation of engineers. They were not version-controlled, so they were scary for new members of the team to modify. We did not use Jenkins beyond these jobs, so it felt foreign to explore or experiment. Jenkins knowledge ended up consolidated in just two senior engineers.
For our new release process, we decided to try GitHub Actions instead of Jenkins. GitHub Actions provides similar job automation but runs on infrastructure managed by GitHub. This infrastructure works near flawlessly because GitHub has a whole team dedicated to maintaining it. Our build and deploy jobs now succeed unless there is a failure that is within our ability to address.
Our GitHub Actions code lives in the same git repository as the rest of our code. Every developer has a copy of it, so there’s a low barrier to entry to engage with it. The code is version controlled, so developers can experiment with changes on branches. Improvements to this code follow our standard development lifecycle, which includes code review and quality assurance testing prior to release in production.
Planning the transition from Jenkins to GitHub Actions was not trivial. Notably, we had to learn how to run our build and deploy jobs on infrastructure outside of our network. We also had to decide how to intuitively trigger these jobs in GitHub Actions.
Running release automation outside our network
Our release automation needs access to infrastructure in our AWS private cloud. Jenkins naturally has this access because we host it ourselves in this same cloud. Since GitHub Actions is hosted externally, we needed to find a way to enable remote access.
We could not simply expose our infrastructure over the public internet: so much publicly exposed infrastructure is a security faux pas. Luckily, our IT team suggested we could remotely access infrastructure through a virtual private network (VPN), similar to how developers connect from their laptops. A quick search turned up a blog post about how to use OpenVPN in GitHub Actions.
We pair-programmed with the IT team to get the VPN working. They configured secrets for a new VPN user: a root certificate, a user certificate, and a user key. We used Ubuntu’s apt-get to install OpenVPN within our job. Eventually, we ran curl on an internal health check endpoint to test if we had remote access. It worked!
As we polished our code, we were impressed by GitHub Actions’ robust job control. If OpenVPN or curl failed, we instructed our job to save the VPN log as an artifact. Since we normally do not watch jobs live, post-mortem access to this log was key to debugging VPN issues.
We were also able to abstract the VPN logic into a reusable composite action. This abstraction not only made the code reusable across jobs but also de-cluttered each caller’s code.
Now, we connect to the VPN with a step like this:
The composite action itself looks like this:
The VPN has worked reliably for our jobs since we completed the transition to GitHub Actions. Though it itself did not make release automation more approachable, it was key to adopting GitHub’s more familiar and reliable infrastructure.
Triggering release automation in GitHub Actions
Most of our Jenkins release automation involved two separate jobs: one that built code and one that deployed code. This separation was intuitive since we’d often build code once but deploy it multiple times: first to an environment for UX review, then to another environment for regression testing, and finally to our production environment.
We initially planned to directly translate these jobs to GitHub Actions and continue manually running them. But, then we realized that we could do better.
We were inspired by our development branch, which diligently receives our latest and greatest code. Before my time, someone had configured this branch to run both the build and deploy jobs in Jenkins when a new commit was merged. Our development environment was always up-to-date, even though we never manually ran jobs. It was so convenient. Could we extend this convenience to the rest of our environments? Yes!
In git, branches are nothing fancy. They’re just named, mutable pointers to commits. And, as our development branch demonstrated, they’re an idiomatic way to track which commit should be released. So, we decided to designate additional GitHub branches as “environment branches”, one per environment. These branches run our build and deploy automation when they receive new commits. To release new code, we update these branches using git push with the force flag.
Environment branches work elegantly within our release process. We start by picking a commit that we want to release, typically from our development branch. We deploy this commit through a series of environments as we gather feedback and check for regressions. Once we know everything looks good, we release to production.
To save time and money across these git pushes, we designed our automation to reuse previous builds.
Environment branches make code releases simple to understand. Developers no longer need to coordinate a build job and a series of deploy jobs; they just git push from one environment to the next. To look at which commits are currently deployed, developers can inspect an environment branch via git log or GitHub’s web interface.
Environment branches empower the whole team to contribute to our release automation code. Developers can make changes on a local branch and try running them by pushing to an environment reserved for experimentation. When the changes look good, they can open a pull request to solicit feedback prior to merging into our shared code.
Conclusion
Our new release automation empowers the whole team to participate in releasing code. The automation runs on reliable infrastructure and a familiar platform. The code is on every developer’s computer and changes to it follow our standard development lifecycle. It’s easy to recommend any developer try picking up a release automation ticket from our backlog or explore their own ideas for improvement. Now, a wider group of us can bring new features and new data online at Measures for Justice.
