Why Data Catalogs Are the Standard for Data Intelligence
How data catalogs can help enterprises looking for the right foundation for data intelligence while boosting employee productivity and the bottom line.
Data culture. Data literacy. Data intelligence. All these things are deemed essential by the market. It’s easy to fall into the trap of feeling daunted, falling behind, or not knowing where to start. If this is happening to you, don’t fret. The truth is, many feel the exact same way and are trying to figure out how to get better at data.
In my role as director of product and cloud marketing, I speak with large enterprises that are making a genuine effort to foster a data culture and use their data more intelligently. Many admit to just starting or being early in their journey. Of course, every organization is at a different stage, and developing a data culture doesn’t happen overnight, so the question becomes, “How do we set the right foundation for data intelligence?”
Data catalogs are the answer.
Gartner positions a data catalog as the foundation “to access and represent all metadata types in a connected knowledge graph.” To illustrate, I’ll share a personal experience about why I think a data catalog is crucial to data intelligence. Some years ago, when I worked at a large global technology company, my manager said, “I want you to figure out what metrics we should measure and tell us if our product is making our customers successful. We don’t have the data or analysis today.” I was surprised. How could that be? How can a successful enterprise not have the data model in place to measure a market-leading product? Have they based their decisions on gut instinct?
As part of my work, I had to create some hypotheses, gather data, analyze it, and create a recommendation. To start, I had to find an expert who had a significant amount of tribal knowledge and could explain what data existed, where it was located, what it meant, how I should use it, and what pitfalls I might encounter when using it. Next, I had to get the data from the data warehouse and write a lot of SQL queries, all while finding the data science people to get their help.
The worst part? Whenever questions about the validity, accuracy, and reliability of the data came up, I had to trust what people told me or rely on my own judgment. After months of working on the project, it turned out to be a worthwhile exercise that delivered insights.
Fast forward to today. After experiencing the capabilities of a proper data management solution, that whole experience would have been much better with a data catalog. Why? Assume the organization had a rich data catalog. I could have searched for and discovered the data myself, understood the context of the data, identified the data’s subject matter experts and asked them questions in the data catalog, written SQL against multiple data sources with active guidance on which data to use and not use, reused existing SQL queries so I didn’t have to write them from scratch, and collaborated with data scientists and others to analyze the data in one place. I would have saved a lot of time and effort. With that saved time, I could have worked on something new and even more valuable for the business.
How does this make the case for why a data catalog is the foundation for data intelligence? It’s simple. If you extrapolate my single experience to everyone who needs or wants to use data for their work, a data catalog makes data more intelligent and useful for a swath of solutions, from enabling self-service for analytics to data governance or cloud data migration. Think about my time spent on the aforementioned project multiplied by thousands of users. That amounts to significant productivity gains.
Data producers can connect data sources and automatically scan data into the catalog. Indexed data can then be organized and governed to comply with data policies. DataOps can develop high-quality data products that allow data consumers to find, understand, and use the data to make faster data-driven decisions that can lead to increased revenue or drive competitive advantage.
Research from the latest Forrester report shows that data catalogs lay a foundation for organizations creating a data culture. The technology drives data use across organizations and enhances data literacy. Organizations with successful data catalog implementations have seen positive changes in the speed and quality of data analysis and in the engagement of teams who need to perform data analysis.
Managing data in the age of big data, data lakes, and self-service is a challenge, and data catalogs are the solution. Data catalogs have evolved from a “nice-to-have” tool to a “must-have” that organizations expect in their arsenal. Without a powerful data catalog, the status quo is impossible to sustain.
About the Author
Jason Lim is the director of product and cloud marketing at Alation. Jason co-founded Koombah, a real-estate startup in China, and AsiaRecon, a technology and innovation tour in Australia. Jason was a contributing writer to Forbes Asia, covering startups and tech trends. Jason is originally from Sydney, Australia and now lives in California. You can contact the author via email, on Twitter, or LinkedIn.