Executive Summary | Unified Platforms for Modern Analytics
Executive summary for the TDWI Best Practices Report: Unified Platforms for Modern Analytics
- By Fern Halper, Ph.D.
- September 10, 2021
The demand for analytics continues to grow. At TDWI we see more organizations looking to advance beyond dashboards to self-service analytics and utilize more sophisticated algorithms such as machine learning (ML). To support this trend, companies are modernizing their data and analytics environment as well. The trend is towards a unified environment that provides an integrated set of tools that address many aspects of the data and analytics life cycle, including data management and development and deployment of sophisticated analytics. The platform is often cloud-based to take advantage of scale and flexibility and support massive amounts of data and compute-intensive workloads. It is automated and augmented to help address data and analytics complexity.
The market is still early for this unified platform although respondents to the survey for this report overwhelmingly agree that unification is an opportunity. The benefits of unification include the ability to capture and leverage emerging and diverse data, achieving faster time to insights, better performance and scalability, and support for advanced analytics. The challenges include vendor lock-in and the politics of ownership.
In this study, respondents identified a minimum set of important features for the unified platform. This reflects today’s market. Fifty-seven percent said data warehouse services needed to be included in the unified platform. Forty-three percent said security services and 33 percent each said data lake and data visualization services. However, respondents all agreed that other services such as data catalogs, cloud services, augmented and automated services, advanced analytics, the ability to support open source and commercial tools, and deployment services were also important.
Organizations are accomplishing or planning to accomplish unification in different ways. On the technology front, a popular approach is to leverage an integrated stack that includes data and analytics capabilities. Another approach is to utilize a data fabric to logically integrate data across hybrid environments and integrate analytics into this fabric. New approaches continue to evolve. On the organizational front, new skills will be needed. For instance, companies will need to hire operations staff to help put analytics into production and monitor its performance—this will be key to success. This includes roles such as MLOps and data engineers.
In addition to new roles for the unified platform, there will be new governance considerations, too. Organizations will need to adopt tools to register and manage models, especially those that will be put into production. They will also need to provide explainability so users can understand output from automated tools. Data and analytics ethics will start to become more important as organizations build and deploy ML models at scale on the unified platform.
This TDWI Best Practices Report examines the adoption, use, challenges, architectures, and best practices for unified platforms for modern analytics. It examines how organizations are using these platforms today and discusses their future plans.
Alation, Denodo, Incorta, Matillion, SAP, SAS, Snowflake, and Wavicle Data sponsored the research and writing of this report.
About the Author
Fern Halper, Ph.D., is vice president and senior director of TDWI Research for advanced analytics. She is well known in the analytics community, having been published hundreds of times on data mining and information technology over the past 20 years. Halper is also co-author of several Dummies books on cloud computing and big data. She focuses on advanced analytics, including predictive analytics, text and social media analysis, machine-learning, AI, cognitive computing, and big data analytics approaches. She has been a partner at industry analyst firm Hurwitz & Associates and a lead data analyst for Bell Labs. Her Ph.D. is from Texas A&M University. You can reach her by email ([email protected]), on Twitter (twitter.com/fhalper), and on LinkedIn (linkedin.com/in/fbhalper).