Data Mesh 101: Data-Driven Transformation
In our world today, when researching Data Mesh, you might have come across phrases like “a sociotechnical approach to build a decentralized data architecture”, data domains, and Data As A Product. The concepts are simple to understand; decentralised, data as a product, and data owners are the ones that know the data best. However, understanding the actions needed to achieve these concepts is harder. By looking at the desired outcome and work backwards, we can understand those steps.
The Goal of Data Mesh
The goal of most companies is to make money and from that, a healthy profit. By providing customers with products and services that they want at the right quality and price point, we can do that. But, this involves the changing needs of the customer and disrupting competitors in a rapidly changing global economy. We can expand the goal to be – meeting the customers in a way that protects or increases our revenue, whilst being able to quickly adapt to disruption, and minimising the cost of operations.
This is much more realistic to how most companies work to fulfil their goals, they want to be agile and create value. This is where agility comes in, as after all, being able to quickly adapt is what agility really is. Having business agility means meeting the customer needs quicker, creating happier customers, but it also means less disruption by competitors. It’s also much cheaper for you as a company, as less time and effort needs to go into changing existing solutions to meet those new challenges, and so the cost of operations reduces. This is the holy grail: happy customers, business agility and cheaper operations, and who doesn’t want that?
Setting up Data Mesh
When you read about Data Mesh you see a conceptualised and very high-level description of a utopian world where all data is shared, and companies can enjoy the benefits of bringing that data together to gain insights or improve operational effectiveness. How though, do we translate that sentiment into something operational and actionable? In a perfect world we would have cross functional teams, publishing data as a product, and an updated data catalogue with information about the data, such as its quality, security and privacy, all things that need to be taken care of, and then finally, where to get the data. The trouble is that many companies will look at data mesh as a technology problem when it’s all about mindset and approach.
Our solution to this is to focus on the process, governance, and mindset from day one. Businesses then can create patterns and reusable assets to compliment how the company will create their data mesh solution. Having the technology part of the solution be out of the box and easily usable means we can focus on the parts that will make the real difference.
How to Make an Impact
Businesses create cross functional teams that include all the roles needed to capture, transform, govern, and publish the data for each data creator. Data-as-a-Product also changes the responsibility model. Source data providers become responsible for the quality of the data instead of centralized Data teams, and this is a fundamental shift that massively improves data quality. Keep in mind that a data consumer can also be a data publisher if they are creating new data from the data they are consuming.
Many companies went agile – small A – by adopting an Agile framework – large A – but they did this without the companywide mindset change required to really have the best impact. Unfortunately, this will be the same for data mesh, it will be seen as a technology problem.
The next challenge- you want to get the benefits without spending a year or more making an infrastructure to support the needed changes to ways of working with data. This again is a challenge and will become more of a challenge over time. Much like with Agile, companies will be inundated with offers for Data Mesh platforms and products, most of them will be tech solutions and centrally managed. This is really an antipattern. Teams need to be able to achieve their goals, adhere to standards and the correct governance, whilst also being able to publish their data. Ideally this needs to be self-service. The next challenge will be – unless you already have a super modern architecture- you’ll have some systems, possibly legacy, that really don’t want to easily give up their data.
The Future
In many ways, data mesh is similiar to agile. For years we have tried to decouple monolithic systems to remove dependencies between systems to reduce the management overhead. The introduction of DevOps teams – further reduces unnecessary dependency management. To us, Data Mesh is the agile transformation of data, we want to decouple data from the systems that create it, organisation’s monolithic systems are now the data platforms many have relied on for years. Data needs to go through the same revolution, it is your gold, it has intrinsic value. Going through this process and continually improving how data flows in your organisation, using DataOps teams, will mean that your operations are cheaper, time to value is vastly improved, and best of all – having business agility means you’re always ready to take on the next competitor.
