Securing the production line

In my previous post, I explained how the security of your production line is intrinsic to the security of the product you’re building. This post picks up on that theme by exploring some practical steps you can take to improve the security of your pipeline.

While each production line has its idiosyncrasies, most modern CI/CD pipelines are fairly similar at a high level. We tend to see a number of core commonalities, such as development environments, source control systems, CI servers, artefact repositories, deployment tooling, etc.

Each of these tools introduces its own set of security challenges. For example, CI servers often store sensitive credentials to access source repositories, artefact repositories, and cloud infrastructure APIs. These tools wield great power; in the wrong hands, that can be quite disastrous.

As Facebook describes of its osquery host visibility tool:

Even when developers think about the security implications of the software they create, it’s important to bake security into the complete software development, build, and distribution pipeline. [..] If we were to assume the codebase itself is secure, the ability to arbitrarily modify the behavior of packaging hosts could still enable malicious behavior. That’s why we also invest in the security of our build and packaging infrastructure.

What can go wrong?

With software updates being ‘weaponized’ to deliver ransomware and backdoored Docker images being used for cryptocurrency mining, there’s clearly no limit to criminal ingenuity… or as Frank Abagnale puts it: “technology breeds crime”. While these stories made the headlines, there are many more that don’t.

The prevalence of insecure build servers publicly exposed to the internet is crying out for attention, much like last year’s series of insecure internet-facing NoSQL databases warned about two years prior. Although NoSQL database insecurity isn’t directly related to pipeline security, it hints at a similar problem – a lot of software tends to be installed, but not correctly configured or operated. Keeping in mind that most software is not secure by default, this can expand your organisation’s attack surface quite significantly.

In business terms, the cost of these issues is often paid in damaged reputations, intellectual property loss, financial loss, and the risk of compromise deeper within the organisation, as CI systems are prime candidates as beachheads from which to launch further attacks.

What can you do about it?

Firstly, don’t run builds on Jenkins master. It’s trivial to circumvent security controls in Jenkins if you’re able to control a build that runs on master, so ensure that all builds execute on slaves. This was demonstrated a few years ago at DEF CON 22.

If you’re still running an old version of Jenkins, consider upgrading. Aside from the importance of keeping in line with the latest security fixes, Jenkins 2.x ensures that security options are enabled by default, meaning that many obvious weaknesses (e.g. default admin credentials, lack of CSRF protection) are avoided in the design and development of the product. This does not mean there’s nothing for you to do – it just means you’re starting out in a much better position.

It’s also worth considering how you can ensure the provenance of the code that runs in production (or gets shipped to your users via app stores). Can you be sure that the packaged artifact you’re deploying has not been modified in any way since it was created? Can you trace it back to a particular tag in your source repository? Can you track all the commits that make up that tag to individual contributors? Can you be sure that those contributors genuinely produced the code within those commits?

One potential solution to address these concerns lies in cryptographic signatures and verifications. GitHub supports GPG signed commits and tags, and similar concepts can be used to sign binary packages that are produced as the result of a build. And don’t forget to validate the signatures.

All the above are fairly specific pointers, but the starting point for many teams is to recognise the build system as a true production system that warrants the same degree of attention as any live environment. That includes monitoring, alerting, patching, secrets management, secure communications channels, access control, changing default credentials, and on the list goes. A threat model goes a long way to identifying many of these issues and ensuring they are handled appropriately in your context.