Executive Summary | Data Science and Big Data: Enterprise Paths to Success
- By Fern Halper, Ph.D.
- December 21, 2016
Big data and data science can provide a significant path to value for organizations. These technologies, methodologies, and skills can help organizations gain additional insight about customers and operations; they can help make organizations more efficient, be a new source of revenue, and make organizations more competitive.
Although many companies are still analyzing structured data, “newer” data sources such as text data, streaming data, and geospatial data are becoming part of an evolving data landscape. However, businesses often struggle with putting an effective analytics and data science strategy together. Part of the issue is with technology. Additionally, there are organizational challenges that must be addressed, which often include building cultures, hiring the right people, and organizing to execute.
TDWI Research finds that there are many paths to value. On the technology front, our survey respondents are utilizing open source and commercial packages to drive big data value. They are using a mix of on-premises and cloud technologies. They are using data warehouses together with newer technologies such as Hadoop and Spark to manage and process big data. They are deploying appliances, MPP (massively parallel processing) databases, and other solutions to meet their big data management needs. A new ecosystem is evolving to support big data and data science.
On the analytics front, data scientists and others needed to succeed in big data are often hard to find. Survey respondents are using different approaches to build data science skills. They are hiring from the outside as well as trying to grow talent internally. They are often looking to business analysts to become more sophisticated analytically to supplement data science expertise. Some are using a team approach. Many organizations are creating centers of excellence to provide analytics and big data expertise and to disseminate learning. Few are hiring chief analytics officers or chief data scientists. Most look to the VP of analytics or the CIO to help them in their efforts.
This TDWI Best Practices Report examines organizations’ experiences with and plans for big data and data science including both technology plans and organizational strategies. It also looks at various big data challenges and how organizations are overcoming them. It examines the importance of new open source models. Finally, it offers recommendations and best practices for successfully implementing big data programs in the organization.
A unique feature of this report is its examination of the characteristics of companies that have actually measured either top-line or bottom-line impact with big data and data science. In other words, it explores how those companies compare against those that haven’t measured value.
IBM, MapR, OpenText, and Snowflake Computing sponsored the research and writing of this report.
About the Author
Fern Halper, Ph.D., is VP of TDWI Research. Her work focuses on AI, generative AI, agentic AI, AI governance, cloud computing, and other modern analytics approaches. She has more than 25 years of experience in data and business analysis and AI and has published numerous articles on data mining and information technology. Halper is co-author of “Dummies” books on cloud computing, hybrid cloud, and service management, as well as Big Data for Dummies. Halper is also the author of the 2026 book, Data Makes the World Go ’Round: The data, tech, and trust behind AI success. She has been a partner at industry analyst firm Hurwitz & Associates and a lead analyst for Bell Labs. She has taught at both Colgate University and Bentley University. Her Ph.D. is from Texas A&M University. You can reach her by email ([email protected]) or LinkedIn (linkedin.com/in/fbhalper).