The State of Data Management
TDWI VP and senior director of research Fern Halper discusses results from her recent “State of Data Management” assessment and report.
- By Upside Staff
- July 28, 2023
In a recent episode of TDWI’s “Speaking of Data” podcast, Fern Halper discussed the results of TDWI’s new State of Data Management report. Halper is vice president and senior research director for advanced analytics at TDWI. [Editor’s note: Speaker quotations have been edited for length and clarity.]
Halper set the stage by defining what TDWI means by the term “data management.” “It’s basically the process of collecting, storing, organizing, maintaining, and using data effectively.” It includes several disciplines, such as data integration and data quality, master data management, data architecture, and metadata management. These disciplines are important because organizations want to make sure data is accurate, consistent, accessible, and secure, and that it can be used to support their needs.
“When we survey organizations,” Halper continued, “we often hear that their biggest need is a solid data infrastructure to support analytics and applications and the ability to monetize data.” To do that, they need a solid data foundation, which is why data management is so important.
To collect information on the state of data management, TDWI created an assessment called the Data Management Maturity Model, available at TDWI.org. It measures five categories of data management: organizational commitment, allocation of resources, architecture, data life cycle, and data governance. The survey, which consists of roughly 50 questions, returns a score out of 100 that corresponds to a specific overall maturity level. The assessment also provides individual scores for the five dimensions.
According to Halper, “what we saw from that data was that the overall average score for 2022 and 2023 was about a 55 out of 100 -- basically right in the middle.” This is what TDWI calls stage three of five, the early established phase of maturity. At stage three, executive leadership is beginning to recognize the importance of data management and they're starting to establish a data strategy and promote data literacy and collaboration across functional silos. Enterprises are also starting to allocate funds to prioritize data management.
“In stage three, we see that organizations are making use of platforms like the cloud and that data integration is becoming critical. We see more reliance on shareable data and the need for metadata and data curation, which is probably why data catalogs are becoming so important.”
The discussion then turned to the challenges organizations are facing with data management. “Respondents say that leaders understand the value of data, but there's a gap between understanding that it's valuable and actually putting support behind the data management efforts.” In this data management assessment, 73% of the respondents agreed that executives value data – yet only about 20% had a strong data strategy in place. In addition, the strategy isn't well communicated; only 12% said their strategy was communicated and only about 40% said a data culture was defined.
“We're also seeing organizations struggle with data literacy,” Halper noted. “That often means understanding the data or, on the analytics front, critical thinking and communicating with data and being able to interpret analytics results.” This often leads to organizations trying to implement data catalogs, which comes with its own set of issues.
“It's often difficult to get people to buy into the catalog approach. People need to be engaged with the catalog, use the catalog, and be committed to updating it.” As an indicator of this difficulty, when asked about maintaining a data catalog, the median score in in the maturity model was a two out of five. When asked about collecting metadata (a component of an effective data catalog), the median score was just one -- the lowest score.
On the technology side, one of the key challenges is data integration. “Close to 50% of respondents in this assessment were still building manual pipelines, which aren't scalable, so that's another problem.” In addition, organizations are saying it’s a priority to democratize their data and make it available to everyone, although most respondents to the maturity model said they only have self-service for some users or some parts of the organization.
When asked about what organizations could do to modernize their data management, Halper had several thoughts. “The first thing is there has to be a business reason [why] you're changing up your data management. If there's a business reason -- for example, to move to the cloud to support modern analytics -- that's going to take you down one path. If your goal it is to reduce the cost of the datacenters, that may take you down another path.”
Halper also reported that 20 to 25% of respondents in recent surveys have some sort of chief data officer or chief analytics officer leading their data efforts, and another 25% have some sort of VP or director of IT filling that role. TDWI research has almost uniformly shown that those organizations that have CDOs or comparable executives tend to be more satisfied with their data efforts, and especially if that executive was part of the C suite.
Halper concluded by noting once a concrete strategy is in place, supported by effective executive leadership, only then should organizations begin to tackle the technological aspects of their data management program, such as cloud solutions, data fabrics or meshes, and other related matters.
[Editor’s note: You can take the assessment or download a free copy of the report here. Listen to the full podcast episode here.]