A case of pandemic agility

How Equal Experts helped HMRC support the economy in four short weeks

In March 2020 the UK went into lockdown, causing the most brutal recession in living memory. The Government had to react quickly, with policies to help the people and businesses in the United Kingdom cope. These include the Coronavirus Job Retention Scheme, the Self Employment Income Support Scheme, and the Eat Out to Help Out Scheme. Once these initiatives were launched, His Majesty’s Revenue and Customs (HMRC) needed to deliver a whole new system, capable of dealing with huge spikes in traffic. And it had to be designed, implemented, and delivered in a matter of weeks. Here’s how Equal Experts helped make it possible.

This case study will help you to understand:

  • How good digital services can be launched to customers within days.
  • Why a digital platform should be treated as a product rather than a project.
  • The importance of culture in a digital platform ecosystem.

Outcomes

67,000

job claims within half an hour of the Job Retention Scheme going live

440,000

people applied for government grants via the Self Employment Income Scheme on the first day

4

weeks to launch all services

About HMRC

HMRC is a non-ministerial department of government. It is the UK’s tax, payments and customs authority, and has a vital purpose: gathering the approximately £788bn tax revenue that pays for the UK’s public services (hospitals, schools, etc.), and helping families and individuals with targeted financial support.

Industry
Government
Organisation size
66k+ employees
Location
UK

Challenge

Working quickly - creating a team to deliver in just four weeks

As the Chancellor of the Exchequer was rethinking the economic structure of the country, HMRC was equally busy gathering together several blended teams of data engineers to support a digital transformation.

A mix of partners – including Equal Experts – enabled HMRC to jointly develop strong cloud technology architecture and improve the diversity of the organisation. They comprised experts technically capable of delivering a practical, straightforward system. A system that needed to withstand demand at levels never experienced before by HMRC.

The scale of the work

HMRC is an established government department, with a multitude of suppliers, hundreds of teams and thousands of engineers.

A huge amount of work was required to build and maintain the platform and services needed to deliver financial support to more than 12 million employed and self-employed workers via the Coronavirus Job Retention Scheme, the Self-Employment Income Support Scheme and the Eat Out to Help Out Scheme.

Due to the timescales, Equal Experts worked with HMRC to enable the team structure to work differently on the Multi Tax Digital Platform (MDTP). Specialists from each discipline needed to be represented at team level. A platform engineer was embedded into each digital service team to safely “short-circuit” existing processes. This allowed us to eliminate risks early in the process and maximise on collaboration.

This was adopted as a short term measure and within weeks of the first launch, we adapted digital platform tools and processes to make some of those “short-circuits” unnecessary.

Solution

The MDTP - how a single cloud platform provides room to build and grow

The UK government’s rapid digital response to coronavirus (COVID-19) was a result of years of investment in people, governance and technology. The Multi-channel Digital Tax Platform is the cloud platform, hosted on Amazon Web Services (AWS), that was home to all the services provided. Originally evolving out of other services and deployed through shared lessons, this provided a “paved road” that allowed our digital service teams to write code to solve these HMRC problems on day one.

Discover more about cloud-based platforms in our Digital Platform playbook.

MDTP is home to:

The cloud platform enabled the HMRC campaign for coronavirus. It held up to record traffic – peaking at 132 million page views in a single week – cementing the importance of having a PaaS in a single public cloud.

The MDTP is a cloud platform for over 244 customer-facing applications and 850 production microservices that have been built to support HMRC’s digital tax strategy. As one of HMRC’s partners working on the MDTP, we’ve helped provide an easy way for teams to build and deploy an application that can scale to handle millions of requests. By building a Platform As A Service (PaaS), or platform-based service cloud computing service, the blended team has enabled HMRC to develop, run, and manage applications without the complexity of building and maintaining the infrastructure.

A Digital Platform is a type of bespoke PaaS: a set of streamlined tools, processes, and people which form platform capabilities to help you and your organisation rapidly meet the needs of your customers.

There are different types of bespoke PaaS, based on workload. The most common types we’ve seen before are Digital and Data Platforms. We advise your organisation to invest in a bespoke PaaS when a clear, homogenous workload starts to emerge across multiple teams.

Equal Experts strongly recommend hosting a bespoke PaaS in a single public cloud. Amazon Web Services (AWS), Azure, or Google Cloud Platform are the main providers. They each offer a tried and tested, on-demand, cloud computing platform, with a wide range of reliable cloud services that can be provisioned instantly and billed on a pay-per-use basis.

The importance of culture in providing focus

The HMRC response would not have been possible without the huge effort of the people who made it happen.

The response relied on digital capability and expertise to make policy into reality quickly – often going from concept to live in just a few weeks.

  • 4 weeks to launch all services
  • 97% customer satisfaction level

Collaborative ways of working were essential. It is difficult to comprehend how impressive the delivery of these services is. Equal Experts consultants were focused on delivery, working alongside a pool of very talented HMRC people. This was a collaborative environment where learning from each other to solve real problems meant that anything was possible.

At Equal Experts, we trade on our ability to learn and share knowledge, rather than protecting or ‘guarding’ it. At its core, Equal Experts is a haven where this sharing happens freely and happily, between like-minded practitioners and our customers.

The established culture and collaboration between digital service and digital platform teams was a foundation to the success of this project. Digital platform teams were already used to bi-directional feedback loops with digital service teams. This meant everyone was able to run these at high-speed, leveraging existing relationships and collaboration tools.

I am in awe of the problem-crunching fury with which HMRC and the Treasury created the furlough scheme and all the other means of support.

Prime Minister Boris Johnson
From the New Deal for Britain speech , June 2020

Collaboration was the foundation to the success

Another shared obstacle was the increase in remote working. In a matter of days, HMRC had gone from being a mainly office-based workforce to having 55,000 people working from home. The most important aspect of delivering a system at speed is the ability for engineers to be able to just ‘get on with it’. The work we’d already done with HMRC supported governance, best practices and a mandate for teams to use certain tools; this meant that any shift made to accommodate this radical change did not risk causing new technology problems or security concerns that might otherwise have delayed delivery.

Automating the setup can have many benefits:

  • Makes an engineer’s life easy by taking away pains of setup
  • Eases the burden of costs for companies
  • The setup itself will become an onboarding document and will be validated with every (new) team member trying to set up their machine
  • Consistent setup across the team and thus helps in the effective pairing
  • Get rid of the “works on my machine” phenomenon
  • Helps in keeping the tools upgraded

Equal Experts already used Slack for effective communication with client teams. It is very easy to have instant conversations but also keep track of what happens elsewhere. A Slack channel could be created ad-hoc for a specific issue. Engineers, analysts and designers from multiple teams and specialisations can focus on collaborating and solving problems without endless meetings or endless email chains.

  • Questions for a specific team – the team uses public team channel as the first option
  • Private Slack channels – for the day to day internal discussions that don’t need to be aired in the public
  • A consistent naming convention, with good channel descriptions.

The other really important aspect of working at speed is that as much as possible can be achieved using self-service. Logs, performance monitoring, automated alerting, a build pipeline that would automatically build and deploy to the non-production environments. It would even configure a Pull Request Builder that would run the tests before committing to the main branch.

A really important aspect of working at speed is that as much as possible can be achieved using self-service. Here is a view from one of the engineers on the project:

“Within next to no time, I had all the supporting components in place. Logs, performance monitoring, automated alerting, a build pipeline that would automatically build and deploy to the non-production environments as soon as I committed to the main branch. It would even configure a Pull Request Builder that would run the tests before committing to the main branch. All this was done in a matter of minutes, giving me time to focus on writing the service itself.”

Adopting a You Build It, You Run It approach

HMRC wanted to give the MDTP team all the capabilities they needed to develop, test, deploy, and operate their own live services. We supported HMRC to adopt a “You Build It, You Run It” approach, something we at Equal Experts use regularly to ‘upskill’ and empower our clients’ teams. This has been a key enabler to supporting so many teams on the MDTP. A Platform-as-a-Service offering maximum operability with minimal friction is easier for teams to operate safely and reliably.

The production support method of ‘You Build It, You Run It’ refers to developers supporting their own production services. You Build It You Run It unlocks weekly or more frequent deployments for Digital Service teams, as it eliminates any operational handoffs. It also incentivises Digital Service teams to balance their time between product and operational features. The net result is more frequent production deployments and more reliable live services.

Find out more about You Build It You Run It.

Understanding when interacting with legacy systems is possible and when it's not

Unless the project is built from scratch, there will always be a requirement to work around legacy systems. In this case, each digital service required interactions with legacy systems across HMRC. With HMRC’s technical experts, we analysed each system to understand whether it was ‘fit for purpose’. Where it was possible to work with the system, we supported service teams to leverage existing, well-defined interfaces to legacy systems. Where there was a potential risk of destabilising the whole process, the platform team quickly and iteratively built well-defined interfaces to new software systems.

Organisations, over the years, build up tens or even hundreds of legacy IT systems (either off the shelf or bespoke). Typically, these systems are stand-alone or have limited ability to integrate with a wider enterprise. A digital platform can allow these disparate legacy systems to evolve into modern-looking, more flexible systems.

Results

Why continuous delivery is an effective, secure solution for change

A digital platform is never “done” – and what has come before is not a predictor of what is to come. Although working to short timescales, this value still stood. Throughout the HMRC project, by adopting a continuous delivery approach, we were able to increase feedback cycles, de-risk production deployments, and ensure production systems could be operated in a reliable functioning condition. This enabled the performance and resilience required so that HMRC could deliver on the Government’s promise to the huge population of claimants accessing the Covid-19 services.

Our approach to Continuous Delivery

  • We start projects by building a “walking skeleton,” automating the path to production
  • This is an end-to-end slice of working functionality
  • It proves the architecture, path to production, and a foundation to build upon, incrementally
  • We then automate the process of building and testing software in a repeatable and consistent way, from development, through test and into production
  • We incrementally adopt continuous delivery and build a deployment pipeline to give us confidence the same software is deployed and tested through each state
  • This confidence allows us to release early and often, with small enhancements directly responding to customer feedback

You may also like

Blog

How HMRC prioritised people, culture, and collaboration – and became the COVID-19 (award) winners.

HMRC

Blog

How Equal Experts helped HMRC to get COVID-19 relief money to people through Continuous Delivery

Case Study

How Equal Experts is helping HMRC to develop the next generation of data engineers

Get in touch

Want to know more?

Are you interested in this project? Or do you have one just like it? Get in touch. We’d love to tell you more about it.