Accelerating software delivery

Technology & Software

Finding the product delivery bottleneck for a large computer product and software manufacturer

Our client creates computer peripherals. Mice, keyboards, headsets, speakers, webcams. To build customer loyalty, and as a valuable support service, they offer a software platform to optimise and customise products.

As millions of customers rely on the software platform, the leadership team recognised the increasing demand for greater throughput and reliability in features. Equal Experts was asked to assess the client’s in-house technology capabilities, validate the strategy for Continuous Delivery, and provide recommendations on accelerating software delivery.

We worked closely with the client on an intense, two-week engagement in November 2020. We held interviews, collected quantitative data, and ran a value stream mapping workshop for all the platform teams.

Outcomes

5 days

to identify the product delivery bottleneck

12 days

to refine the strategy for continuous delivery

About the client

Our client is a manufacturer of computer peripherals and software, with global headquarters. It’s one of the world’s leading manufacturers of products such as keyboards, mice, tablet accessories, speakers, and webcams.

Industry

Software

Organisation size

7000+ employees

Location

USA

Length of project

Two weeks

Challenge

Understanding how to double deployments

Our client’s leadership team wanted to speed up product features and reduce error rates for their customers. The answer was to move the platform from monthly to fortnightly deployments. While some of the technology challenges needed to achieve this were well understood, it wasn’t clear in what order they should be tackled. The product delivery bottleneck wasn’t obvious.

To get a clearer picture, we collected deployment data, and visualised lead time (time from code to deployment) versus interval (time between deployments). The lead time showed build, test, and deploy activities always happened in under 14 days. The interval revealed planning and development took one to two months, which told us the product delivery bottleneck was in there.

Through in-depth interviews with platform team members, we learned that deployments were scheduled months in advance, in lockstep with hardware launch dates. This meant each version was a large batch of new features and could go no faster than peripheral launches.

Now we had identified planning lockstep as the product delivery bottleneck, we needed to demonstrate how removing it would unlock other improvements in the deployment pipeline. To accelerate deployment and ensure features could be enabled on the right date, we recommended the use of feature toggles.

We then worked with the relevant teams to visualise GitLab CI metrics in Grafana dashboards. This provides an automated monitoring solution for speed of deployments, builds, and code across all software services in the organisation.

Solution

Mapping out a plan for improvement

Next, we needed to plan a prioritised roadmap of improvement efforts.

The platform teams were included in the creation of the improvement roadmap from the outset. This gave the leadership team confidence that there was buy-in for inserting a raft of improvement efforts into the product roadmap.

Inspecting all aspects of feature development, we considered the entire feature lifecycle. This led us to an emphasis on ways of working. We advocated experimenting with work-in-progress limits, designing smaller feature definitions, and pairing during code reviews.

To reduce error rates and their impact on customers, we also looked at technology changes in the platform itself. This could be done by incrementally migrating the frontend to TypeScript, and creating a canary beta channel for faster customer feedback on errors.

Results

Refining the Continuous Delivery strategy

Before our arrival, the client had drafted a preliminary Continuous Delivery strategy. There was a focus on accelerating developer feedback loops, by speeding up the build system and the automated tests. This preliminary strategy needed to be pressure tested, using our previous experiences of Continuous Delivery.

The platform build system was already being migrated from a data centre to AWS. Working with the platform teams to achieve earlier payoffs, we pinpointed which tasks to pull forward in the migration plan. This included decreasing network latency by consolidating AWS regions and reducing compilation times by replacing Git submodules with internal libraries.

Another area for improvement was automated testing cycles. These were complicated by the hundreds of different peripherals supported by the platform. By examining the USB bridge connector to the customer’s device lab, and where different types of testing occurred in the deployment pipeline, we outlined how to create a device simulator for peripherals, and how to re-sequence device lab testing. This would allow developers to receive feedback in minutes, rather than hours.

As a result of this refinement and recommendations, our client is now in a strong position to achieve a step-change in how its software services are built, tested, and deployed to customers.

Working with this client was a pleasure. There was a strong sense of collaboration, and together we established a technology roadmap that will help the organisation to make a big leap forward.

Steve Smith

Continuous Delivery & Operability Practice Lead , Equal Experts