Stop wondering and start using probabilistic forecasting: A case study at John Lewis & Partners

Probabilistic forecasting predicts outcomes in the future based on events from the past. Much has been written on the theory of sampling historic data thousands of times to build predictions, but there aren’t many evidence based examples of where probabilistic forecasting has been used effectively in a client environment.

Knowing that this could be a great way of examining estimation cost and opportunity benefit data in a deadline driven business domain, we were recently able to demonstrate its impact through practical application at John Lewis & Partners (JL&P). The JL&P website agile teams were under pressure to provide data that would inform long-term vision for the business, but they didn’t have a low-impact way to create scientifically backed forecasts around future delivery capabilities. Frustration was building as onerous processes started to look like the only way forward on reporting.

What is probabilistic forecasting?

An accurate estimate

It’s probably easier to start with what probabilistic forecasting is not. A common misnomer is the idea of an “accurate estimate.” It’s not about forecasting the exact number of items a team’s work system can produce. It’s more about finding a predictable variance of the system that doesn’t incur wasteful and costly estimation work. Accuracy in this context refers to a lower variance range.

For example:

• Overly precise forecast: We will deliver these exact 20 items on 19 January at 12:00.
• Accurate forecast: We can deliver this feature between 15 December and 1 January with an 85% confidence
• Inaccurate forecast: We’ll deliver that feature between 3 and 60 weeks from now.

Making agile technology work smarter, not harder

At the time, it looked like the best option at JL&P was to use detailed hours-based estimation to answer the questions posed to agile teams, a costly and time-consuming task. There was also a perception among some stakeholders that those questions could not be answered in the Partnership’s Kanban environment. The cost of estimation – for just the team we worked on – was looking like 10% of the team’s annual capacity. That’s a lot of potential development time spent on pure estimation. Assuming a heavy estimation process was rolled out to all web teams, the business was potentially looking at a huge increase in overheads.

Something Equal Experts is known for in the tech industry is our ability to look outside the box for creative ways of working to solve complex problems differently. As part of our embedded partnership, JL&P wanted us to help them build on the solid foundation put in place by their teams to diversify and evolve their strategic thinking. This meant sharing what our own experiences told us was possible with a mature Kanban team.

We wanted to see if there was a more efficient way to provide a fact-based and accurate delivery forecast. We know that predictability improves through doing work rather than talking about work; building incrementally allows us to spot potential scenarios and predict outcomes quicker.

The Partnership was already using contemporary agile techniques – they just needed support to leverage those and free them up from burdensome processes. Rather than hours-based estimations with narrow insight, we used probabilistic forecasting to answer key business questions:

1. When will the work be done?
2. How much has been done, and how much is left to do?
3. What is the probability of success?

Answering these questions with the scientific evidence that comes from multiple data calibrations gives the business greater confidence to make good decisions around the viability of a piece of work.

The benefits of probabilistic forecasting in a deadline driven environment

Here’s a good illustration of the benefits of probabilistic forecasting at JL&P, in terms of hours spent on a single estimation session using a current popular method, versus time spent on Monte Carlo, the probabilistic forecasting solution introduced into the client team:

Depending on the size, experience and estimation frequency of the team, this ratio can lead to a 5 – 10% annual capacity consumption for a single team, simply to obtain estimates. The Probabilistic forecasting tool (Monte Carlo) creates reliable forecasts at a fraction of the effort of traditional methods. It also has the added benefit of obtaining “estimate input data” from existing team processes, reducing the frequency or eliminating the need for expensive estimation sessions.

Forecast results

All forecasts represented in the table below were obtained without a single estimation session. Existing team processes and historic data were used to obtain the input data for the Monte Carlo Simulations.

Out of two large, complex initiatives that required fixed date deliveries in the same week, the team delivered both on time.

Conclusion

Data from the last 2 years of using probabilistic forecasting in JL&P’s complex, deadline-driven environment shows extremely high accuracy compared to effort expended on estimation. It also highlights the potential waste of time and team capacity spent on heavy estimation sessions that don’t predictably lead to “better answers.”

Our highly collaborative ways of working meant we were able to coach engineering teams, to free them up from onerous and costly admin processes, instead facilitating the adoption of the light-touch, low-cost and high accuracy techniques of probabilistic forecasting.