Data Mesh: Benefits, Challenges, and How To Get Started
Fern Halper, TDWI’s vice president and senior director of research for advanced analytics, discusses the data mesh, including the opportunities and challenges it presents.
- By Upside Staff
- August 9, 2023
In a TDWI “Speaking of Data” podcast earlier this year, TDWI’s Fern Halper explored the data mesh -- a new approach to managing today’s increasingly complex data landscapes. Halper is vice president and senior research director for advanced analytics at TDWI. [Editor’s note: Speaker quotations have been edited for length and clarity.]
“The idea was introduced in 2019 by consultant Zhamak Dehghani,” Halper began. “When she introduced it, she said it wasn’t an architecture as such but rather a new framework.”
The framework was built on four pillars:
- Domain-oriented ownership. “By this, she meant that businesses own their own data, which means that ownership is decentralized,” Halper said. As part of this, Dehghani also envisioned that there would be an increase in data sharing across the various business units to avoid unnecessary silos.
- Data as a product. Although there has been much talk of data products lately, this aspect of the data mesh essentially means domains making their data available to those who need it and in such a way that users are satisfied with it. The data should be high quality, understandable, and interoperable.
- Self-service. “The key here,” Halper said, “is to allow multiple personas -- from data scientists to business users -- to make use of the platform and access the data the domains are serving up.”
- Federated governance. In the data mesh, governance is managed in part by team members from across the business domains. This federated model balances the autonomy of the business domains with the need for compliance, interoperability, and security.
With these four pillars, the expectation is that business decision-makers will have faster access to data and analytics with which to drive the business. However, when asked whether organizations were adopting this framework, Halper said TDWI research hasn’t shown much uptake yet.
“In 2022, we did a survey and as part of it, asked how many were using it. Less than 15% said they were and an additional 18% said they were planning to. About a third said they weren’t planning to use a mesh but they agreed with its principles. Then a third said they had no plan to implement a mesh, period.” Halper pointed out that CEOs and CDOs especially seemed to fall into the third that had no plans but liked data mesh principles, perhaps because they thought it would make it easier to manage the domain-oriented, federated structure the mesh recommends.
Survey respondents liked other features of the data mesh as well. For example, most of the pillars involve streamlining user access to the data they need -- domain ownership, data as a product, and self-service. Respondents also reported that, even in cases where users were having to wait for domains to package and provide data, there was an increase in data literacy.
However, organizations have found just as many reasons not to use the data mesh. For example, a primary concern is that the domains that own the data won’t actually share it, creating unnecessary silos. Another concern is that the decentralized nature of the data ownership will make governance harder, including change management.
“There are also organizational concerns about whether people will lose their jobs as a result of data and analytics moving to the domains,” Halper explained. If the organization is moving from a centralized data function, what happens to those people when data moves to the domains?
Do organizations interested in trying the data mesh need to jump in all at once?
“I don’t think it has to be all or none,” Halper said. “It depends on how they’re organized initially. For instance, if everything is centralized, they can possibly move some of the data to the domains and start there.” Halper also offered the example of a company that likes the idea of data as a product; some of its domains can begin offering data as a product to other domains.
“It’s a matter of weighing costs versus benefits and where your organization is now,” Halper added. “Moving from a centralized model to a decentralized model may make sense for organizations with multiple business units, but it can be difficult for others.”
As for the future, Halper doesn’t see significant change in adoption in the next year or two. “We may see more organizations using some of the principles, especially because we’re seeing a lot of activity around data products.” In fact, according to the most recent TDWI Teams, Skills, and Budgets Report, some of the biggest budget increases were in areas related to data products. Even so, Halper added, even if people do adopt the mesh, it’s just as likely that they will pick and choose from among the various aspects and tweak it to make a framework that works for them.
[Editor’s note: To hear the full conversation, replay the podcast episode here.]