The solution: Becoming a platform team

Working with civil servants to understand the needs of government change programmes, we developed a value stream model of demand for the platform – backed up by online dashboards – to understand bottle necks both in process and technology. We also reviewed support issues, using no-blame retrospectives to understand root causes and implement measures to prevent them reoccurring. 

The team developed a vision for new ways of working, with four major initiatives: 

• Empowering the teams developing APIs and SFTs to build and deploy their services without manual support from the platform team.  
• A programme of platform simplification to reduce the number and variety of component versions running and make them more evergreen.
• Improving observability.
• Reducing infrastructure costs by removing inefficient processes and leveraging cloud infrastructure to scale and switch off services when not required. 

BECOMING A PLATFORM TEAM

The first step was to establish consistent and reactive support for the application developers building services for the platform. We created a front door process to manage requests consistently, enabling customers to see where their request was, and application developers and platform teams to self-organise in response to changes in demand data. We experimented with various team structures and delivery life cycles as we learned.

A major learning came at the end of the 2022 Financial Year when our dashboards showed a big spike in demand. Multiple programmes were all targeting deliveries at the same time.  As our work was typically at the end of the delivery lifecycle, we could predict big demand peaks, so we started producing heat maps showing which periods were likely to be busiest and used them to smooth out peaks and maintain flow. 

SIMPLIFYING THE PLATFORM

It was clear that organisational changes alone would not provide the level of service we needed, so the team established a quarterly platform optimisation process to review performance and set measurable goals for the next quarter. 

We focused on simplifying the platform and reducing errors, following three core principles: 

• Bring all assets under source control
• Reduce the number of components used and the complexity of implementation
• Ensure the platform was observable 

An example was certificate expiry, which was a major source of live issues. We found thousands of certificates within the infrastructure and logged each one, enabling them to be renewed before they expired. 

There were multiple versions of the platform running, with accompanying support and infrastructure costs; there was also a risk that older versions of the platform would eventually contain insecure or obsolete components. We worked with HMRC product teams to migrate these to the latest version, allowing us to retire older systems. 

Monitoring and alerting was improved. First, we increased the amount of logging platform components performed. We then created consistent patterns for storing log data, making them searchable so that teams could create alerts and dashboards relevant to their services. 

EMPOWERING ENGINEERS

As the platform became simpler, our next goal was to empower teams to deliver without needing the Platform Team to intervene. Using the ‘Paved Road’ approach, we identified the most frequently requested support ticket types and automated or handed over these processes. This reduced dependencies between teams allowing application developers to deliver at their own pace, and freeing up platform teams to focus further on improving the platform. 

A major impediment was the build pipeline. All teams used a single pipeline containing complex business logic. This made it difficult for engineers to manage the pipeline as they couldn’t know if code they wanted to change was required by other teams. We prioritised simplifying the build pipeline, forming a shared team of platform engineers and application developers. This enabled each application development team to own their own deployments and adapt them for their application. 

OPTIMISING

As platform stability improved and delivery tasks were handed over, platform engineers were able to switch focus from reactive support to proactive maintenance. Logging and monitoring was expanded with alerts created not just for failures, but also to detect pre-failure conditions so that engineers could intervene before an issue became critical.  

Infrastructure management was introduced, (e.g. switching off non-production environments at night, switching off under-used instances, and auto scaling). This allowed system costs to match service demand better.