State of Data Report Emphasizes Emerging Shift to a Decentralized Model
Second annual study, commissioned by Starburst and Red Hat, highlights increasing data access demands and flexibility challenges.
Note: TDWI’s editors carefully choose press releases related to the data and analytics industry. We have edited and/or condensed this release to highlight key information but make no claims as to its accuracy.
New market research commissioned by Starburst and Red Hat uncovers that 55 percent of organizations claim the pandemic has made data access more critical, a slight increase from the 2021 study. Enterprises plan to prioritize multicloud flexibility and ease of use when it comes to selecting data infrastructure solutions.
The second annual report, “The State of Data and What’s Next,” conducted by independent research firm Enterprise Management Associates (EMA), found that the shift to quick and flexible deployments is imperative for driving the business functions and insights required to deliver valuable customer experiences in today’s fast-paced, distributed environment.
The new research indicates that surveyed global organizations are increasingly relying on data, with pressures for data access growing to meet customer demands in an advancing digital, highly mobile landscape. It points to a major disconnect between the critical demand for faster data access and having a centralized data strategy, indicative of three main challenges facing data teams today:
- Data sprawl has grown in complexity: Survey respondents reported that their organizations currently use an average of 4-6 data platforms (43 percent), with an average of 11 percent employing 10-12 Furthermore, organizations are adding new data types to their environments at an increasing rate, citing streaming data (65 percent of respondents) as the most popular data type they plan to collect in the next year, followed by video and event data (60 percent of respondents).
- Pains in the pipeline process persist: In the wake of the COVID-19 pandemic, organizations must now accelerate data-driven decision-making to keep pace with ever-shifting customer However, for over 48 percent of survey respondents, it takes more than 24 hours to create a data pipeline, then another 24 hours (for 51 percent of respondents) to move data pipelines into production, making real-time business operations a challenge. This, combined with the need for faster data access, is pushing the industry away from the painful pipeline process and into a more decentralized model, or data mesh.
- AI/ML are increasing and placing greater pressure on a variety of systems: EMA is seeing a shift wherein data science (ML and neural networks) is rated as the most important analytics workload, which is applying increased pressure on already complex data platforms. Organizations need to process vast amounts of AI/ML data to fuel these workloads, but 31 percent of respondents said that data constantly being moved and changed makes finding it difficult. Along with the struggle to find data science resources, more automation of data science workloads and better data access are required to improve AI/ML models and reduce resource
The research findings also show that enterprises are turning to specific practices and product capabilities to meet these challenges:
- Prioritize faster data access: As a result of the challenges over the last two years, organizations must become nimble in providing fast, reliable access to data anytime, The survey results show that demands of customer experience (33 percent), the ever-growing challenge of staying ahead of risk and market swings (29 percent), and employee engagement (29 percent) are the driving factors for these shifts.
- Shift to a more decentralized model: Centralizing data certainly holds benefits, such as consolidated cost, a high level of control, and ease of However, centralization also comes with increased risk, with a single point of failure and limited flexibility. This lack of flexibility can leave a business slow to adapt in a rapidly changing environment.
- Automation of critical technology systems: With data spread throughout organizations, the number-one challenge enterprises face is the increased complexity of a hybrid, multicloud environment (40 percent). Organizations are looking to automation to maintain a competitive edge in the industry (38 percent), with a focus on the underlying data infrastructure with the automation of data pipelines, the adoption of intelligent search (33 percent), and the implementation of AI and ML in data processing to drive business decisions (32 percent).
- Multicloud flexibility: Respondents noted that in 2021, 56 percent of their data was in the cloud compared to 44 percent on premises. When asked the same question this year, respondents stated that 59 percent of their organization’s data resides in the cloud compared to 41 percent on premises. The growing move to the cloud points to why multicloud flexibility (43 percent) has the most impact on buying decisions with hybrid interoperability making a significant jump to 34 percent, up from 26 percent in 2021.
“Customers creating AI/ML enabled applications must rely on accessible data in order to accelerate model development and the deployment of intelligent applications across hybrid multicloud environments,” said Steven Huels, senior director, software engineering, Red Hat. “By creating a foundation for data and applications on cloud architecture, developers and data scientists can more quickly and repeatedly meet their business goals through the delivery of data-driven, intelligent applications.”
For more information on the data mesh model, visit us here. You can also download the full report.