Why Most Data Platforms Fail (And How You Can Succeed)
The co-founders of Monte Carlo discuss the challenges -- and solutions -- to operationalizing data at scale.
- By Barr Moses, Lior Gavish
- May 24, 2021
Many data engineers and analysts find themselves part of a familiar story. They've been hired by a company that's trumpeting an investment in building a "data platform," given access to a patchwork suite of tools and data sets, and then have innovative projects relegated to the low spot on priority lists. How can they break this cycle?
Twenty years ago, you'd be hard-pressed to find a company that employed more than a handful of data analysts -- if that. In 2021, analyzing data is one of the world's hottest professions, with data scientists, engineers, and analysts embedded in nearly every functional area of the modern enterprise, from marketing to customer service, and for good reason.
From Nike's investment in customer data and predictive analytics to centuries-old banking firms such as UBS using AI to detect fraud at scale, business leaders are sold on the concept of disrupting or revolutionizing their legacy industries through data.
Although these companies have found ways to identify use cases and solve specific problems, this model -- embedded data teams solving business-unit-level problems -- isn't always scalable or sustainable.
As more teams across the company become capable of working with data, redundancies creep in, work diffuses, and infrastructure maintenance costs may skyrocket.
We share common challenges faced by companies when building data platforms and highlight a few best practices for overcoming them:
Working in Silos
Silos happen, especially during rapid growth.
When employees set out to solve a specific problem for their specific team, they likely aren't considering how the solution they build fits into the company's overall ecosystem. Those team-level blinders lead to silos, where different teams doing similar work within the organization don't communicate with or learn from each other.
Working in silos can allow data scientists to move faster, but over time, the silos get in the way of aligning around shared goals and core values about the characteristics of a good data platform. Silos can also lead to bad habits of "empire building" -- employees or teams trying to gain authority or clout by hoarding resources or access to data.
To break down these barriers, start by making sure you understand the full landscape of how data currently operates within your company and who is responsible for those processes. Talk to the data engineers and scientists within core areas and identify possible redundancies, lost efficiencies, and missed opportunities that you can solve with better collaboration.
Start to build a community of practice among these business users. Getting your data analysts and engineers connected now will help streamline their adoption of your future platform.
Employing an Outdated Approach to Data
Just because your CTO uses a buzzword (such as "data-driven") doesn't mean the executive has a clear understanding of how data functions in your organization -- not to mention the data literacy required to understand, create, and transform data assets. Without this knowledge, it's unlikely your CTO will want to invest in your vision of a data platform.
In some cases, leaders may also be reluctant to stop relying on legacy software instead of investing in cloud-based technologies or a more modern data stack. They fall prey to what behavioral economists call the "sunk-cost fallacy" -- the misguided insistence on continuing to invest in something that isn't working simply because you've already been investing in it.
In larger companies, you may encounter a cultural aversion to risk-taking; leaders may prefer to continue doing what has worked in the past instead of taking on a bold new initiative such as operationalizing a companywide data platform.
In either case, helping management understand the bigger vision and potential value that modern data platforms can add is key to gaining stakeholder buy-in. If the data consumers within a company -- the business leaders who put data to use -- aren't sold on the idea of a data platform, they may not support changing the processes their teams have built.
Start small, using case studies (internal or external), lunch-and-learns, and other material to encourage discussion and brainstorming about how investing in data could positively impact your business. Calculate the tangible costs of sticking with outdated technologies. Keep your discussions concrete; share realistic estimates of time or resources your stakeholders can expect to invest and which key metrics will be impacted.
Lack of Clear Ownership and Buy-In
Saying you want to build a data platform and actually building one are two entirely different things. More often than not, unless your company is hiring an entire data platform team, the impetus and motivation behind building one often falls to a few excited data engineers and a director or two.
In addition to building a platform that serves higher-level needs, consider tailoring the project to support adjacent functional areas by baking in features that provide upfront value. For example, find out how much time a certain team spends by tackling ad hoc requests from their own stakeholders or determine how having a data platform will increase their leverage and productivity.
Both leadership and grassroots buy-in are critical. To get it, you must tie the platform back to how it will help meet not just your company's goal (i.e., lower costs, higher customer retention, or faster growth) but also the needs of your fellow data practitioners.
Security and Compliance
Security and compliance concerns will be another technical challenge to any new platform. The more you work to centralize and increase accessibility to data, the more concerns people will raise about ensuring security. Investing in a data catalog or lineage solution will get you part way there, but often these tools lack automation, are built in silos, and don't leverage metadata in a use-case-specific way.
To start, bake in a security-first approach to your platform with appropriate privileges, retention policies, and protocols. Plan to implement data observability and monitoring so you can understand the health of your data assets at all points, from ingestion to analytics. Detecting and identifying errors or anomalies will help ensure security and build trust in your data.
Moving Forward
Implementing a scalable (and effective) data platform requires patience, determination, and (as with most difficult challenges) taking it one small step at a time.
Your data platform will only scale if you communicate its value to your organization both now and in the future. To be successful, develop a comprehensive but realistic strategy around executing this vision.
About the Authors
Barr Moses is CEO and co-founder of Monte Carlo, a data reliability company and creator of a data observability platform. Previously, she was VP customer operations at customer success company Gainsight, where she helped scale the company 10x in revenue and, among other functions, built the data/analytics team. Prior to that, she was a management consultant at Bain & Company and a research assistant at the statistics department at Stanford University. She also served in the Israeli Air Force as a commander of an intelligence data analyst unit. Barr graduated from Stanford with a B.Sc. in mathematical and computational science.
Lior Gavish is CTO and co-founder of Monte Carlo, a data observability company. Prior to Monte Carlo, Lior co-founded cybersecurity startup Sookasa, which was acquired by Barracuda in 2016. At Barracuda, Lior was SVP of engineering, launching ML products for fraud prevention. Lior holds an MBA from Stanford and an MSC in computer science from Tel-Aviv University.