Want to know more?
Are you interested in this project? Or do you have one just like it? Get in touch. We’d love to tell you more about it.
The power of automation in data certification at the John Lewis Partnership
Streamlining Tableau Data Source certification for efficiency
The Partnership Data Platform (PDP) at the John Lewis Partnership relies on Tableau, a visual analytics platform, to help the organisation solve problems using data. As a centralised group responsible for the quality of all Tableau assets, the Tableau team was facing challenges scaling the service they provided to the Partnership, with delays in the provision of crucial insights to decision-makers in the business.
The solution was to establish a ‘Centre of Excellence’ to enable scalable self-service capabilities that would empower teams working on the PDP to take ownership of the implementation and maintenance of their own Tableau data sources, whilst maintaining strong governance. Working closely with the new Centre of Excellence, Equal Experts supported the Financial Services (FS) team to help finesse the delivery process, developing a suite of automation tools and an optimised certification process to significantly reduce time to market and – most importantly – time to value for the business. The FS team was chosen as it was one of the first teams delivering products on the new platform, so it was a great opportunity to learn through doing.
The work establishes a paved road that will serve as a blueprint for other PDP teams to replicate this success.
Key outcomes
90%
Reduction in time spent on manual checks
40%
Reduction in deployment times
100%
Ability to scale a consistent process that improves the quality of maintained Tableau data sources
About the client
The John Lewis Partnership is one of the UK’s oldest, largest and most popular retailers; it includes John Lewis and Waitrose and is the largest employee-owned business in the UK. There are 34 John Lewis stores across the UK, and johnlewis.com; with 329 Waitrose stores across the UK, and waitrose.com. It has total trading sales of £12.4 billion, a workforce of 74,000 Partners (employees).
Industry
Retail
Organisation size
74,000 Partners
Location
UK
Length of project
2+ years
Challenge
Cutting complexity and boosting efficiency for better data insights
The Tableau Certification process was created to address inconsistencies in the way that Tableau Data Sources were created and maintained within the Partnership. With few checks in place when Tableau was first brought in as an analytics tool teams ended up creating Tabelau data sources inconsistently. This meant that trustworthy/reliable data was hard to discover. The Tableau team addressed this issue by creating a change request process that defined a gold standard for data sources. When ownership of the Tableau data source was transferred to the PDP Standing Teams the process was not conducive to modern engineering practices; with several redundant steps and unnecessary handovers creating complexity. Manual processes with slow feedback cycles meant certification timelines could be lengthy and not in sync with how a modern engineering team works. In addition, the Tableau Centre of Excellence lacked the engineering capacity to build and maintain the tooling required to automate and streamline the processes.
Solution
Scaling automation for a more efficient workflow
The PDP FS team worked closely with the Tableau Centre of Excellence to overhaul a highly complicated, manually driven maintenance process that suffered from multiple hand-offs and slow feedback cycles.
The team developed automated tooling to support certification, data refresh and data quality alerts, and releasing a Tableau Data Source into Production; this significantly reduced manual effort and increased process efficiency for Data Engineers and Business Owners.
Using programmatic approaches to leverage metadata in Tableau’s API, the team discovered helpful information which allowed them to perform complex tasks easily, including analysing the impact of proposed changes ahead of implementation, to enable more informed decision making.
Most excitingly, we’ve demonstrated that processes we’ve wanted to implement for a long time could be automated, developing four critical automation tools that transformed both the productionisation and change request processes:
- Audit Tool: automatically analyses and reports on the health of Tableau Data Sources, ensuring they meet compliance standards before being productionised. This innovation significantly reduced (by 90%) the time spent on manual checks. Previously this was a 17 step checklist that had to be manually completed each time a certification was needed. Improvements include better governance through automation and a repeatable process that enables incremental testing and faster delivery.
- Data Refresh Automation: keeps Tableau Data Sources consistently up-to-date as new data is ingested in real-time, eliminating the need for manual data refreshes and vastly improving query time and reporting performance. This has ensured continuous availability of current data, leading to more accurate and timely decision-making. There is also a significant cost reduction in data warehouse compute time as the data is cached in Tableau.
- Promotion Tool: facilitates the seamless promotion of Tableau Data Sources from Sandbox (Dev) to Release and finally Supported (Production), adhering to rigorous DevOps release processes. This enables seamless switching between production and non-production data for testing, reduced errors, increased consistency and a more efficient release process.
- Data Quality Tool: automatically exposes data quality issues so that reporting users can be warned when working with stale data, providing transparency and immediate feedback on data quality issues.
Technical overview in a nutshell
Results
Efficiency and agility through automation
The collaborative approach taken by the PDP Financial Services team helped cement the value of Tableau Data Source certification across the business. They provided hands-on examples and practical guidance to demonstrate how the process could be effectively managed:
- Workflow optimisation: a streamlined process to reduce confusion and improve transparency in task management.
- Jira Integration: to enable better tracking of certification tasks.
- CI/CD deployment pipeline: to automate many of the manual steps involved in certification.
- Feedback loop: for faster feedback cycles, allowing teams to quickly address any issues or non-compliance discovered during the certification process.
- Documentation and knowledge sharing: for best practice.
The team demonstrated the value of the work in a practical use case which involved centralising insurance product data received from an external vendor, which previously relied on manual spreadsheet reviews. Improvements to data ingestions, tool integration and compliance has showcased:
- Enhanced decision-making: the commercial team now accesses accurate and up-to-date complaints data, significantly improving decision-making capabilities.
- Reduced cost: since we only refresh the Tableau Data Source when there is actual data to refresh (instead of running on a fixed schedule), we can reduce warehouse compute time costs.
- Optimised Tableau usage: users across different functions, including self-serve use cases, now rely on a central, certified source for complaint data, ensuring a single version of the truth.
- Agile release processes: collaboration with the Tableau Centre of Excellence team improved the agility of the Tableau Data Source maintenance release process, reducing deployment times by 40%.
Conclusion
The Tableau Centre of Excellence has improved efficiency by reducing feedback loops and automating costly manual steps, significantly decreasing the time required to deliver changes to production. In the future, the team plans to adopt the use of a workflow tool, ensuring everyone is aligned and the process is transparent.
The new, streamlined process is now scalable, enabling other teams to adopt these best practices and tools, thereby standardising the certification process across the organisation.