Rethinking production systems

Amidst the talk about pipelines and production systems, and despite the parallels drawn between production lines in manufacturing and software engineering, the industry seems to have lost the true meaning of ‘production’. Reflecting on this leads us back to a clearer separation between production and the product.

In this post I’ll look at why pipeline security matters, and it’s followed by a second post on the practical steps you can take to improve the security of your pipelines.

The dictionary definition of ‘production’ is ‘the action of making or manufacturing from components or raw materials, or the process of being so manufactured’.

However, if you ask almost any software delivery team, they’ll tell you that ‘production’ is where the system runs with real data and real users. ‘Production’ is an environment; in fact, it’s the last in a series of environments (such as development, testing, staging, etc.) that are all used to deliver software.

However, the side effect of this is that we can easily forget to treat the production line as a production system.

Pipelines are production systems… literally!

In the most literal sense, pipelines are production systems. And despite what we might call it, ‘production’ is not an environment, it is a process. Of course, the product we build is critically important – it’s the reason we’re there in the first place. But it’s vital that we don’t lose sight of the value we gain from our delivery pipelines, and the implications that has for us in how we operate them.

For example, how many build pipelines have you worked with that were properly monitored? How many pipelines have their logs and metrics fed into ELK or Grafana, or produce reliable audit events to trace activity in the various stages in the production line? How many pipelines have you worked with that permitted any authenticated user to run any job? How many even required authentication?

Pipelines are intrinsic to the product

Well-managed pipelines are intrinsic to high quality, secure software. For example, we can’t produce reliable performance test results unless we can be confident that our performance testing environment accurately reflects production. By the same token, we can’t produce secure software if we can’t trust the security of the process by which it was produced. 

A good example of this can be found in food safety, where product packaging contains clear warnings about allergens – particularly those that can mean the difference between life and death. It’s one thing to say ‘may contain traces of nuts or peanuts’; but it’s an entirely different situation to offer a ‘nut free’ product. Both of those statements are as much about the production process as the product itself, and both matter a great deal to the end user.

Securing the production line

Life and death isn’t always at stake; however, there are many other factors to consider. Build pipelines produce the software that runs in the live environment, and often have the ability to deploy directly to that environment. Failing to adequately secure such powerful tools leaves us open to costly compromise.

With all the talk of GDPR, it’s easy to forget that personal data isn’t the only thing worth protecting, nor is it the only thing attackers are interested in (as Tesla found out when its exposed Kubernetes cluster was abused for cryptocurrency mining).

So how do we go about securing the production line? A good starting point is to threat model our pipelines in the same way we do for the software we’re building. A well-constructed threat model will highlight areas of risk that need our attention and drive prioritisation of any improvements, leading to in an increasingly secure pipeline based on risk.

Unless we can provide a production line that is secure enough for our needs, we cannot build software that is secure enough for our needs. For some practical steps you can take to secure your pipelines, read my second post.