Data
Make better decisions with data
Data is so important to our clients, and at the same time the technology and tooling changes so rapidly. The landscape available to data practitioners is changing all the time. We ask ourselves: What architectures should we use? What ways of working should we adopt? What new tools have we found useful? What looks great but actually gets in the way of making data useful?
These are our recommendations and insights into what we think you should use, explore or avoid when it comes to all-things data.
Data Ops
Challenge
Organisations struggle to use their data to drive decision making. Some common scenarios we see include:
- Lack of trust in the data. Either information is out of date due to persistent failures in the flow of data or the data does not reliably represent what is actually happening in the organisation.
- Inability to change the data. Existing data pipelines or reports cannot be changed for fear of breaking the existing information, so insights fail to keep pace with the changing organisation.
- Inability to access the data. Data is locked away or held hostage in current operational systems and cannot be accessed to provide insight and to aid decision making.
Recommendation
Adopt and use modern engineering practices when building data pipelines to manage the flow of data, so that pipelines are reliable and can be changed safely and frequently.
Data pipelines are no different from any other kind of software and benefit from the same practices that have been proven to accelerate software delivery in other areas: infrastructure as code; configuration management of the pipeline; continuous integration and deployment; working in small batches; test driven development and monitoring and observation of the pipeline at various stages.of the pipeline at various stages.
These practices have been shown to increase the ability to deliver software and to reduce rework. But in the data world, pipelines are often not developed and supported using these best practices, leading to failures in data pipelines, long lead times to make data accessible to end users and ultimately loss of trust in the data. Adopting modern engineering practices in a data space, such as DataOps, means that you can make data flow to people who need it more reliably, and create new flows more quickly, improving your ability to understand your business operations and customers.
Resources
Equal Experts Data Pipeline Playbook
Expert Talks Online – Principles, practice and pitfalls for creating Data Pipelines
Ml Ops
Domain-driven Pipelines
Multi-Variant
Hyper-scale SQL Data Pipelines
Explore
Organisations struggle to use their data to drive decision making. Some common scenarios we see include:
Avoid
Organisations struggle to use their data to drive decision making. Some common scenarios we see include: