Want to know more?
Are you interested in this project? Or do you have one just like it? Get in touch. We’d love to tell you more about it.
Rebuilding the plane mid-air at HMRC
How a shift to platform thinking drove efficiency during digital transformation
This case study examines how Equal Experts supported a shift to platform thinking, helping HMRC to protect critical national services during a comprehensive programme of digital transformation and innovation.
HMRC’s Enterprise Integration Services (EIS) delivery group is responsible for building secure integration between HMRC’s internal systems and external organisations, including other government departments. The department supports 100s of billions of transactions every year.
Equal Experts was asked to take ownership of rearchitecting the Container Managed Application (CMA) platform within EIS; the goal was to reduce both the costs of managing the platform, and the time taken to deploy changes. The CMA hosts all the integrations and APIs for the Borders and Trade, and Customs Declaration Services (CDS), both critical systems that control the UK border and ports. If they fail, trucks are left queuing at the border and food and medicines go undelivered.
Outcomes
3
months to take over from outgoing supplier following a discovery
50
support tickets per quarter, down from 750
About HMRC
HMRC is a non-ministerial department of government. It is the UK’s tax, payments and customs authority, and has a vital purpose: gathering the approximately £788bn tax revenue that pays for the UK’s public services (hospitals, schools, etc.), and helping families and individuals with targeted financial support.
Industry
Government
Organisation size
66k+ employees
Location
UK
Length of project
Two years (ongoing)
Challenge
Joining the dots to reduce cost and facilitate change
The CMA platform supports key services such as HMRC’s Borders and Trade systems, and is essential to the smooth running of UK imports and exports. These are busy change programmes with frequent releases of new services. The platform was maintained and developed by a mixed team of supplier consultants (EE and ecosystem partner Esynergy), civil servants and a small number of individual contractors working directly for HMRC.
The platform hosts APIs and Secure File Transfers (SFTs) which enable HMRC’s many systems to communicate with each other. It had been built at pace to meet the needs of EU Exit, but was complex and difficult to run.
The key challenges were:
- Infrastructure costs were high – services were running when not required and some services were over-provisioned.
- API and SFT deployments required manual build, configuration and release steps that both increased risk and created dependencies between the platform and application development teams.
- Deployment pipelines for every service were entangled – too complex to be managed by a single application team.
- Observability and alerting was limited, with no single screen to monitor the platform’s health.
- Systems were tightly coupled, making proactive maintenance and upgrading ageing components difficult.
Solution
Becoming a platform team
Working with civil servants to understand the needs of government change programmes, we developed a value stream model of demand for the platform – backed up by online dashboards – to understand bottle necks both in process and technology. We also reviewed support issues, using no-blame retrospectives to understand root causes and implement measures to prevent them reoccurring.
The team developed a vision for new ways of working, with four major initiatives:
- Empowering the teams developing APIs and SFTs to build and deploy their services without manual support from the platform team.
- A programme of platform simplification to reduce the number and variety of component versions running and make them more evergreen.
- Improving observability.
- Reducing infrastructure costs by removing inefficient processes and leveraging cloud infrastructure to scale and switch off services when not required.
The first step was to establish consistent and reactive support for the application developers building services for the platform. We created a front door process to manage requests consistently, enabling customers to see where their request was, and application developers and platform teams to self-organise in response to changes in demand data. We experimented with various team structures and delivery life cycles as we learned.
A major learning came at the end of the 2022 Financial Year when our dashboards showed a big spike in demand. Multiple programmes were all targeting deliveries at the same time. As our work was typically at the end of the delivery lifecycle, we could predict big demand peaks, so we started producing heat maps showing which periods were likely to be busiest and used them to smooth out peaks and maintain flow.
Simplifying the platform
It was clear that organisational changes alone would not provide the level of service we needed, so the team established a quarterly platform optimisation process to review performance and set measurable goals for the next quarter.
We focused on simplifying the platform and reducing errors, following three core principles:
- Bring all assets under source control
- Reduce the number of components used and the complexity of implementation
- Ensure the platform was observable
An example was certificate expiry, which was a major source of live issues. We found thousands of certificates within the infrastructure and logged each one, enabling them to be renewed before they expired.
There were multiple versions of the platform running, with accompanying support and infrastructure costs; there was also a risk that older versions of the platform would eventually contain insecure or obsolete components. We worked with HMRC product teams to migrate these to the latest version, allowing us to retire older systems.
Monitoring and alerting was improved. First, we increased the amount of logging platform components performed. We then created consistent patterns for storing log data, making them searchable so that teams could create alerts and dashboards relevant to their services.
Empowering engineers
As the platform became simpler, our next goal was to empower teams to deliver without needing the Platform Team to intervene. Using the ‘Paved Road’ approach, we identified the most frequently requested support ticket types and automated or handed over these processes. This reduced dependencies between teams allowing application developers to deliver at their own pace, and freeing up platform teams to focus further on improving the platform.
A major impediment was the build pipeline. All teams used a single pipeline containing complex business logic. This made it difficult for engineers to manage the pipeline as they couldn’t know if code they wanted to change was required by other teams. We prioritised simplifying the build pipeline, forming a shared team of platform engineers and application developers. This enabled each application development team to own their own deployments and adapt them for their application.
Optimising
As platform stability improved and delivery tasks were handed over, platform engineers were able to switch focus from reactive support to proactive maintenance. Logging and monitoring was expanded with alerts created not just for failures, but also to detect pre-failure conditions so that engineers could intervene before an issue became critical.
Infrastructure management was introduced, (e.g. switching off non-production environments at night, switching off under-used instances, and auto scaling). This allowed system costs to match service demand better.
Results
After 18 months of stabilisation and optimisation, significant changes had been brought to the platform. Quarterly support tickets were reduced from 750 to 50. Change Delivery teams were empowered to manage their own build and deploy processes and a £1.4m reduction in infrastructure costs was identified. Equal Experts supported HMRC to maintain all deliveries on schedule, delivering process and cost improvement without any additional funding.