If you’re happy with your company’s data analytics capabilities and you’re using data to drive business decisions effectively, you can stop reading now.
Unfortunately, few are in this category. In our experience, even companies with sophisticated data architectures sometimes struggle to use their data. The following three scenarios may sound painfully familiar.
Company A: Sophisticated, but Reaching Limits of Current Technology
Multiple domain-specific applications stream data into an ‘extract, transform, load’ (ETL) pipeline then into a data lake. A team of highly skilled data engineers analyses the data and generates reports for various internal consumers. The engineering capabilities and data lake infrastructure are impressive, but the data analytics system is becoming a victim of its own success. Useful reports simply spur demand for even more reports. More domains want to pump data into the lake, and the variety and number of requests keeps growing.
The data analytics system cannot scale, so lead time for new reports increases and report quality declines. Shadow data analytics takes hold, with reports created by data consumers who cannot wait for the data team. These consumers get their data from other sources, causing a security and governance nightmare.
For Company A, a data mesh would be an effective, low-cost solution. It siphons off routine reporting tasks, saving the data lake and skillful engineering effort for analytic work.
Company B: Data Silos Pose Barrier to Growth
This company has a collection of domain-specific applications for HR, Legal, Sales, Shipping, and other departments. Each application has its own data store constructed using various technologies. These include relational, noSQL, and graph databases, as well as flat files and event streams stored on-premise and in public cloud.
While each domain-specific application adequately reports on its own data, Company B struggles to generate actionable insights by combining data across domains. In this case, a data mesh would be perfect for aggregating data products generated by each domain in ways that provide holistic business insight.
Company C: Small but Growing Fast
Company C is not yet using on-demand reporting to drive business decisions. Reports might be generated manually at regular intervals, or on an as-needed basis. This approach can work in the early phases of growth, but demand for actionable data will soon grow faster than the ability to generate it.
Building a simple data mesh would be an inexpensive way for Company C to enter the realm of on-demand analytics and reporting. It would deliver immediate benefits then grow and adapt in response to future demands. The initial investment paves the way for more substantial investments in traditional data architectures if that becomes desirable.
What is a Data Mesh?
In any of the above scenarios, a data mesh may provide immediate value and a foundation for future growth. However, it can be hard to understand exactly what a data mesh is and how it might benefit your company. It is not a product. You cannot download data mesh software. While there is a common set of tooling used to build data meshes, there is no one-click install.
Simply put, a data mesh is an architectural pattern which operates much like microservices. It’s often described as an almost revolutionary approach, using self-service platforms to embrace the distributed nature of data. Yet it uses the same industry-standard tools and techniques that have been used at scale by microservices for almost a decade.
We view data mesh as an evolutionary application of widely used techniques and technologies, not a risky rewrite of data analytics architectures. A small data mesh can be up and running in a short period of time. Maintenance is often limited to adding new data products or altering others if the underlying data structures change. In either case, risk is reduced because changes are made by domain experts familiar with the data.
Data Products and Data Nodes
Other terms linked to data mesh include ‘data products’ and ‘data nodes’. Data products encompass the code, data, and infrastructure required to deliver a domain-specific data structure that produces business value in a self-service manner. They can be used for analytical and operational purposes and may be relatively simple (for example monthly sales by office). However, much of the time smaller, simpler data products are combined into a more complex data node, which can be thought of as a read-only microservice.
As is often the case with microservice architectures, each node uses its own preferred technologies and techniques to meet specific requirements. Also like microservices, the technologies and techniques used by each node are often subject to some degree of engineering or architectural governance. The data mesh architecture does not mandate means of communication or API standards, but nodes often employ RESTful APIs documented through tools like Swagger as well as various streaming techniques.
Data mesh topologies can quickly become complex as more varied types of data are added. We recommend starting small and letting your data mesh evolve with demand. The actual topology of mesh nodes will depend on your company’s unique needs and its layout of data sources. Obviously, this topology will change over time; the data mesh architecture allows for that.
The data product concept is central to data mesh architectures. Each node exposes a collection of data products to the end user as well as to other data products in the mesh. In this way, each data product can become a building block for larger and more useful data products. When data products build blocks into arbitrarily complex structures, one of the most useful aspects of a data mesh is born: customised cross-domain data products (see Figure 1).
Highly customised data capabilities arise when one data product incorporates other data products into itself. In Figure 1, the Finance Node retrieves data from the Sales Node, combines that data with its own, and returns the resulting data product to the data dashboard.
Ten Advantages of Data Mesh
1. Better Scaling Capabilities
The scenarios described for Companies A, B, and C are typical. They are not the result of bad engineering or insufficient budgets; they are not even a drawback of traditional data architectures. We believe anti-patterns like these arise from an attempt to use traditional data architectures to meet today’s varied demands for data and analytics.
The following sequence illustrates how data lake architectures can be overwhelmed as demand grows.
Stage One
The below configuration is often successful initially. The number of data providers and data consumers is low, and the specialised data team can keep up with the supply of data and the demand for reports and analytics.
Stage Two
In the second phase, success stimulates more demand for data insights. The number of publishers and subscribers grows, and workload on the data team increases. However, the architectural and organisational model does not scale, so delays occur.
Stage Three
Eventually pipelines are overwhelmed. There is a growing backlog for adding new data sources and generating new reporting insights. Long wait times often lead to shadow data analytics, where teams turn to unofficial sources of data. This can result in severe security and governance issues.
2. Domain Ownership of Data Encouraged
Traditional data analytics attempts to combine data from multiple sources into a single monolithic data lake or data warehouse. With data mesh, domain experts own the data and package it in ways that maintain integrity and fitness for purpose.
A drawback with traditional ETL pipelines and data analytics architectures is that data engineers are unfamiliar with the data they process. They are not domain experts and do not have ownership of data. When pipelines are not processing much data, this is not an issue. Data engineers tend to be highly intelligent and can easily learn about the small amount of data they process. But as demand escalates, problems can begin. The volume of data challenges even the most talented data engineering team. A common symptom of this is longer lead times to generate even simple reports coupled with lower quality of the data included. The distributed nature of data mesh helps eliminate this problem. Domain teams own their data and control how it is exposed to the rest of the system.
Domain-driven design can play an important role in defining corporate domains, and well-defined domains are critical to implementing an effective data mesh.
3. More Flexibility, Less Dependency
As with microservices, tight coupling is the enemy of a highly functional data mesh. The ‘independently deployable’ rule for microservices also applies here; every node on a mesh should be deployable without making corresponding changes to other nodes in the mesh. In the Figure 1 set-up, a deployment to the Finance node should always be made by itself, independently, without a corresponding deployment to the Sales node, and vice versa.
Coordinating deployments of individual data mesh nodes can be difficult. Adhering to the independently deployable rule often involves the application of a versioning scheme to data products, with each domain responsible for its own pipelines. Distributed pipelines tend to eliminate the tight coupling of ingestion, storage, transformation, and consumption of data typical of traditional data architectures like data lakes.
The design influence of microservices on a data mesh is apparent in its flexibility; it can expand and contract to match data topology as it grows in some areas and shrinks in others.
4. Better Data Quality and Discoverability
A data product is the smallest deliverable unit a data mesh can provide. Because it is created by domain experts who own the data, the quality tends to be much higher than data provided by other architectures.
Data products must be easily discoverable to maintain the usefulness of the mesh. Most implementations will use some sort of data catalogue or registry of data products. To make the catalogue useful, information like description, owners, component data products, and lineage are all included and often updated as part of a build pipeline.
The docs-as-code pattern is an excellent practice to employ when building a data mesh. Quick and accurate discoverability is key: if a data product’s metadata is allowed to become out-of-date, the usefulness of the data mesh will dwindle.
5. Better Alignment of Data with Business Needs
Distributed data mesh nodes can call one another just like microservices call one another. Together they can generate and collate data products from multiple sources to deliver on-demand actionable reporting. A data mesh can even be used to increase observability of a company’s activities – a simple use case would be ordering more inventory depending on an analysis of sales patterns over the last few days. Though this is not a typical use of a mesh, it shows the flexibility of the solution.
Figure 2 illustrates how data products in various domains can call other data products and integrate the responses into their own offerings. This capability promotes code reuse and allows arbitrarily complex data structures to be composed with relative ease.
6. Each Data Node can Scale as Required
Some nodes on a data mesh may provide important data products but are accessed infrequently. Others may provide constant access to event streams. Each data node needs to scale independently, as per microservices in a microservice architecture. Independent scalability of application components is a core tenet of cloud-native applications.
Container management tools like Kubernetes that automatically create, delete, and scale data nodes fit well in a data mesh architecture, using pods to house data products. As Figure 5 illustrates, the Kubernetes horizontal pod autoscaler (HPA) can monitor resource demand and automatically scale the number of pod replicas, which independently scales copies of any one data node. By default, the HPA determines whether scaling up or down is necessary every 60 seconds. When changes are required, the number of replicas increases or decreases accordingly.
7. Governance Implemented at Data Product Level
In a data mesh, governance is federated within each data product. It’s best to have the most knowledgeable team responsible for implementing governance; this reduces centralised bottlenecks like data review meetings. Although some bottlenecks can be avoided, be aware that in most cases federated governance places an extra demand on the domain team.
8. Industry Standard Tooling can be Used
Data meshes are built using well-known microservice technologies, patterns, and DevOps pipelines that have been heavily used in industry for years. They employ tools and principles commonly used in microservice architectures such as containers, Kubernetes, service meshes (Istio, Consul, Linkerd), and zero-trust security measures such as continuous verifications, identity-based segmentation, least privilege principle and automated context collection and response.
Container management tools like Kubernetes or service meshes like Istio have tens of thousands of successful installations. RESTful APIs and streaming have been in use for over 20 years. The risk profile of adopting a data mesh is comparable to adopting microservices. It is not a new technology; it is a new way of approaching data.
9. Direct Communication Reduces Risk of Misunderstanding
A data mesh architecture forces domain experts to speak directly with data consumers. This helps avoid misunderstandings which might otherwise lead to errors in reports and/or delays in their generation.
Those who best understand the data own it and compose it into data products for others to consume in a self-service way. Unlike a small team of specialised data engineers struggling to keep up with ever increasing supply and demand, domain experts understand consumers’ needs and build data products to meet them.
10. Scale According to Demand
A company can begin with a small data mesh of just a few nodes and see value very quickly. Though a data mesh can be thought of as microservices for data, the reality is a little more complex. With the principles described in this article, you’ll be able to start your mesh small and let internal demand drive growth. Figure 7 shows how it can evolve from two domains and a few data products to a company-wide architecture.
Please get in touch if you’d like to discuss the concepts outlined in this blog.
Mike is a Managing Principal Consultant at Sourced Group. Over the last 25 years Mike has worked as an engineer, architect, and leader of large engineering organizations.
