Our Thinking, Tech Focus Fri 17th November, 2023
Maintenance mode survey confirms traditional solutions just don’t work
I speak with customers and consultants across the Equal Experts network, to help our customers solve scaling problems and achieve business agility. I recently surveyed 40 organizations on how they implement maintenance mode, and the answers show what we suspected – traditional solutions don’t protect business outcomes.
In Why developers doing maintenance mode is best for your business, I outlined the Maintenance Mode Problem that many organizations eventually face. Who should maintain and run your non-differentiating services, once they are live and reach zero demand? You’ll want to reassign teams to new propositions and resize teams to reduce costs, but how do you do it without sacrificing service reliability, future feature delivery, or job satisfaction levels?
I recently gave a keynote talk at DevOps Enterprise Summit Las Vegas 2023, to answer that question. The video is here.
In the talk, I described a recent survey of 40 organizations in the Equal Experts network, in which we asked about the traditional maintenance mode solutions:
- Solution #1 is for delivery teams to build new services in the foreground, and maintain their own zero demand services in the background, while an operations team runs all zero demand services
- Solution #2 is for delivery teams to build new services, while an operations team maintains and runs all zero demand services
We ran the survey because we wanted to better understand the business impact of those solutions. Here’s a summary:
Solution #1 – delivery teams maintain their own zero demand services
This is delivery teams building new services in the foreground, and maintaining their zero demand services in the background. An operations team runs all live services. Here’s a composite retailer to illustrate this, with non-differentiating teams maintaining their own zero demand services.
We believe this solution is inadequate. You can’t reduce costs, teams have little intrinsic motivation, and delivery lead times are longer as one team with two services will have two business owners prioritizing two different workstreams. Our survey responders agreed with our hypotheses.
52% agreed that feature delivery was slower after maintenance mode began. A delivery lead said ‘there was no governance, teams were drowning in background maintenance work’. Someone else complained ‘business owners keep trying to squeeze features into our Friday maintenance days’. An engineer said ‘feature requests were blocked because our manager didn’t want to change a service already in maintenance mode’.
58% said job satisfaction was worse than before. A delivery manager said ‘we’ve had to slow down work on the new service, to dial up maintenance on the old service, and it’s had a big impact on team morale’.
And 100% of responders saw no cost reductions with this solution. We expected that result, as delivery teams maintaining their own zero demand services doesn’t create many resizing opportunities. It’s why our customers often believe this maintenance mode solution isn’t value for money.
Solution #2 – operations team maintains and runs all zero demand services
This is delivery teams building new services, while an operations team maintains and runs all zero demand services. Here’s the composite retailer again, with zero demand services transitioned into the operations team. Delivery teams can then be reassigned to new services, resized, or retired as necessary.
We often see this solution in the wild, and we believe it’s systemically flawed. Transitioning everything to an operations team can’t assure your existing outcomes. An operations team usually lacks the time/skills to maintain zero demand services, a clear mission to inspire job satisfaction, or unambiguous business ownership to prioritize future delivery. Our survey responders agreed with us.
55% agreed that service reliability worsened with this maintenance mode solution. An engineer said ‘we once had an early morning deployment that failed, and it required 20 hours of remedial work because the Ops team hadn’t been able to do any updates for a year’.
55% said feature delivery was slower than before. A delivery lead lamented that ‘the ability to handle maintenance is a painful concept here, there’s little or no allocation in operations contracts for upgrade work, and there’s a lot of waste in the process’. Another said ‘one team, many services, many business owners, doesn’t make for fast decisions’.
And 88% saw a negative impact on job satisfaction. An engineer said ‘the Ops team was permanently unhappy and I felt bad for them’. A tester said ‘there’s little sense of mission or purpose for any of our teams’.
Conclusion
Our survey of 40 organizations across the Equal Experts network has shown the traditional maintenance mode solutions don’t work. They’re popular because they’re easy to implement, and they can achieve your new capacity and cost outcomes. But they can’t assure existing outcomes, and there’s a substantial negative impact on your organization.
We offer to our customers an alternative maintenance mode solution called multi-product teams, which is based on You Build It You Run It principles and the idea that teams need clear business ownership. We want to put our customers in the best position to succeed, and when it comes to zero demand services, the traditional solutions just aren’t good enough.