Data and Analytics Leaders Report Wasting Funds on Bad Data
Study says 85 percent claim flawed data management leads to poor decision making and lost revenue.
Note: TDWI’s editors carefully choose press releases related to the data and analytics industry. We have edited and/or condensed this release to highlight key information but make no claims as to its accuracy.
As enterprises fiercely compete for data engineers, a new global poll out from Wakefield Research and Fivetran, provider of automated data integration, shows an average of 44 percent of an engineer’s time is wasted building and rebuilding data pipelines that connect data lakes and warehouses with databases and applications. This significant outlay of human capital is troubling: 71 percent of respondents say end users are making business decisions with old or error-prone data -- with 66 percent saying their C-suite doesn’t know this is even happening. As a result, 85 percent of enterprises have made bad decisions that have cost them money.
Companies are paying large sums to reach these bad outcomes. Data and analytics leaders report a median of 12 data engineers earning an average of $98,400 a year each. Spending 44 percent of their time away from working on advanced models and analytics adds up to $520K a year.
“What we’re seeing from this study is that data and analytics leaders are really struggling to keep up,” said George Fraser, CEO of Fivetran. “It would be one thing if the processes companies used for manually building and managing pipelines were optimized, but 80 percent of those surveyed admit they have had to rebuild data pipelines after deployment -- because of changing APIs, for example. For 39 percent, they say this happens often or all of the time.”
In addition to being error prone, the process for deriving perceived value from the data also moves at a snail’s pace. Only 13 percent report being able to derive value from newly collected data in minutes or hours. Instead, for 76 percent of companies, it takes days or up to a week to prepare the data for revenue-impacting decisions, including 74 percent of companies with $500 million in revenue or more.
In other findings:
- Two in three (69 percent) of data and analytics leaders say their business outcomes at their company would be somewhat or significantly improved if their data team was able to contribute more to business decisions rather than manual pipeline management
- A large majority (90 percent) of those polled in the U.S. say hiring staff would be a major component if they could effectively scale up data management capacity, compared with 74 percent outside the U.S.
- Nearly all (97 percent) say business outcomes would be improved if their data team could spend more time devoted to the analytics behind data-driven business decisions
For additional insights and images from The State of Data Management Report, please visit the Fivetran blog.