The importance of user involvement in MLOps

When using MLOps, it’s easy to focus on the technical aspects of the project. But as we explain in our recently published Playbook, Operationalising Machine Learning, it’s vital that MLOps is based around user involvement at every stage – including some employees you might not expect.

Back in 2014, tech giant Amazon built an internal ML University so that its in-house developers could keep their skills up to date. In 2022, developers still use the university – but so do product managers, program managers and a host of other business users from across Amazon.

What Amazon realised was that giving novice users an understanding of basic AI and ML ideas empowered those users to get involved with data teams, and resulted in better projects. Business users at Amazon play a collaborative role in developing a strong business case for ML models, driving solutions that will meet the needs of the business and its customers.

Without this collaboration, ML teams risk building impressive prototypes that never get business buy-in, or don’t have real world customer impact. That’s something to think about considering that IDC reports that 47% of AI projects never get past the initial experimentation phase, and 28% projects simply fail.

The importance of user involvement in MLOps

MLOps can help the success of AI projects by providing a structured framework for moving ML models from development through to production and management. Where and how should data teams start to build user involvement into this process?

Here are five ways you should involve end users in your MLOps process:

Step 1: Ask users for input before development starts

A common pitfall when surfacing a new machine learning score or insight is that end users don’t understand or trust new data points. This can lead to them ignoring the insight – no matter how useful it is.

This can be avoided by involving users at the very start of an ML development. What problem does the user expect the model to solve for them? Use this insight to guide the initial investigation and analysis.

Step 2: Demonstrate and Iterate

Once development starts, make a point of demonstrating model results to users as part of the iterative model development – take users on the journey with you. This is an opportunity to gain early feedback that can help guide development of models that will deliver real benefit to the business and its customers. Data teams should surface ML models for early feedback from users before full productionisation. Tools such as Streamlit and Dash can help to prototype and share models with end users.

Step 3: Focus on explainability

As the model nears completion, ensure that you have something that can be explained – this may be the model itself, or how it arrives at a recommendation or insight.

If you’re building a model that will provide an insight into a credit risk score, you might need to explain what data is being used to drive the insight, and how this insight can be applied within the business user’s regular process of processing a loan application, for example.

Step 4: Monitor your users’ experience

Once a model is live, make sure users are involved in testing, and can provide feedback on any bugs or faults they experience. Consider also using telemetrics for monitoring, so that you can monitor performance of the model and be alerted in case of any issues. You should consider sharing these metrics with business users where appropriate.

These steps will help to build and maintain user trust in the model, and increase the likelihood that the results generated by ML will be adopted as intended.

Step 5: Adopt continuous improvement

When an ML model is in production, you will almost certainly continue to improve the service throughout its lifetime. To maintain high levels of user involvement, capture iterations of your service as versions, and help users to migrate to newer versions.

It’s important to provide good, current user documentation and regularly test how models appear from the user’s perspective. Finally, when you retire a service, have a clear process and ensure that users are supported if a model will no longer be supported.

Summary

We believe that ML services should be developed and treated as a product, meaning organisations should apply the same behaviours and standards that would be used when developing any other software product.

When developing an ML model, it is essential to identify, profile and maintain an active relationship with the end users of your ML service. Work with users to identify requirements that feed into your development backlog, involve users in validating features and improvements, and notify them of updates and outages. In doing so, you will secure buy-in from business users and increase the odds of the AI project delivering real business value.