Paving a road for incident response

At the time, John Lewis & Partners and Equal Experts were building the John Lewis & Partners Digital Platform (JLDP), to handle the ever-growing number of product teams and digital services. A digital platform is a collection of paved roads, providing fault-free and frictionless user journeys for product teams.

It was important to add an incident response paved road to JLDP. To be comfortable with the You Build It You Run It principle of on-call product teams, engineers needed incident response to be streamlined. The existing workflow involved multiple handoffs:

1. An analyst created a ServiceNow ticket, started a private chat room, and phoned an application support analyst.
2. The application support analyst worked to resolve the incident, and phoned an incident manager for major incidents.
3. The incident manager coordinated urgent response efforts between multiple teams, and handled stakeholder communications.

A fully automated workflow was implemented for digital services in JLDP, which used the PagerDuty incident response platform to connect microservices with Slack and ServiceNow. JLDP automatically provisions a new digital service as an on-call policy and team rota in PagerDuty. When an alert is fired:

1. PagerDuty automatically creates a ServiceNow ticket, and calls the on-call product team engineer immediately.
2. The product team engineer creates a public chat room, works to resolve the incident, and contacts an incident manager for major incidents.
3. The incident manager coordinates urgent response efforts between multiple teams, and handles stakeholder communications.

Incorporating as-is incident management was vital. John Lewis & Partners had incident managers who were skilled facilitators and communicators, and they lightened cognitive load for on-call engineers during incident response. Incident managers were modelled as a PagerDuty on-call team rota, and could be phoned during an incident with a single button click in PagerDuty.

The incident response paved road was a key step in the operability journey for John Lewis & Partners. Replacing spreadsheets and phone calls with PagerDuty alerts reduced Time To Acknowledge (TTA) from 5-20 minutes to a consistent 60 seconds. Switching from private to public chat rooms allowed anyone to learn from response efforts, during and after incidents. Implementing bi-directional sync between PagerDuty and ServiceNow improved data capture quality in incident tickets, and allowed customer service teams to contact product teams by simply raising a ServiceNow ticket.

Assessing on-call product teams

Initially, You Build It You Run It caused some concerns from product team engineers, who had not been on-call before. Understanding and overcoming their fears, by openly discussing and addressing them, was just as important as changes to processes and tools. The operating model now receives very positive feedback from product teams, who are constantly learning by building and running their own digital services at scale.

The rate of deployments has dramatically increased, and in-incident financial losses have fallen. A good example of this was Black Friday 2020, which saw record user traffic due to the COVID-19 pandemic. The johnlewis.com website performed well, and online sales were 50% higher than 2019.

In 2021, John Lewis & Partners and Equal Experts ran a cost/benefit analysis on two years of data, which validated the value proposition of You Build It You Run It. In deployment throughput, product teams averaged one deploy every 3 days, whereas the application support team averaged weekly deployments with high variability.

In service reliability, on-call product teams had fewer incidents with a financial loss than the third party managed service, and their time to restore was much faster. 44 digital services had a Mean Time To Recovery (MTTR) of 27 mins with low variability. The application support team had an MTTR of 1 hour with high variability.

The table below shows that You Build It You Run It was superior in terms of scale, throughput, and reliability. It includes a revenue protection effectiveness measure, which was devised for a fair comparison between the two operating models despite their differing teams, services, and ranges of financial exposure. Revenue protection effectiveness was defined as the % of expected revenue loss per incident that was realised, based on the time to restore. You Build It You Run it was more effective, year on year.