Succeed at Your First Contact with Big Data and Hadoop
To overcome barriers and get started with big data and Hadoop, you need the right business case, the right training, and a good starting point.
- By Philip Russom
- June 30, 2016
For years, we at TDWI have been using surveys (and other research methods) to gauge users’ progress with Hadoop, big data, advanced analytics, and related issues. Our two largest surveys on these subjects are the surveys for the 2013 TDWI report Integrating Hadoop into Business Intelligence and the 2015 TDWI report Hadoop for the Enterprise.
I cannot help but notice that the top two barriers to working with Hadoop and big data (as seen in surveys for both reports) are:
#1: Inadequate staffing or skills for big data analytics -- 62 percent of survey respondents in 2013; 42 percent in 2015
#2: Lack of a compelling business case -- 40 percent of survey respondents in 2013; 31 percent in 2015
Both surveys also garnered significant responses for “not enough information on how to get started” as a potential barrier. Please allow me to share a few tips for coping with these barriers.
A Compelling Business Case
Why embrace big data? For one thing, big data and Hadoop are here to stay. Eventually, big data will be assimilated and we’ll just think of it as a special category of enterprise data. Likewise, Hadoop will become a common data persistence platform alongside other forms of open source software and the many brands of relational databases available from vendors of enterprise software.
For you and your employer to stay caught up -- to stay competitive, to improve decision making, to enhance your view of customers and operations, to support new data-driven business goals, to handle modern data types and volumes -- you need to take that journey with big data and possibly Hadoop. You need to get appropriate skills and get started integrating big data and Hadoop into enterprise solutions so that later you can use them with new applications in analytics and modern data-driven operations.
That’s a compelling business case, but so is the following. Big data usually comes from data sources that are new to you, and new data almost always leads to new insights, new ways to optimize operations, and new dimensions for the customer experience.
For example, many of you are contributing data to the bespoke “complete view of the customer.” New big data can add more fields and dimensions to that view for a better understanding of the customer, which in turn has benefits in customer retention and growth.
Training to Close the Skills Gap
Returning to the surveys I mentioned, a “skills gap” is the top barrier seen in survey responses. One way to close that gap is with training.
TDWI has over 20 years of experience as a provider of vendor-neutral, user-oriented best practices in data-driven disciplines, and we continually update our training curricula to cover new and emerging best practices, such as those for big data, advanced analytics, and Hadoop. For training on specific vendor tools, you should contact those vendors. Of course many data professionals are also capable of training themselves and learning on the job.
Related to skills is the question of hiring. Many data management teams are frustrated by the lack of hirable professionals for emerging platforms and disciplines, such as big data, advanced analytics, Hadoop, and data science.
TDWI has seen many organizations vault this hurdle by starting with experienced data professionals (either as established employees or new hires) and training them in the new areas. This is natural, given that many data professionals are already cross-trained in multiple data fields (e.g., data integration, quality, warehousing, reporting) and most enjoy learning new skills for personal edification and as valuable items on their resumes.
A Valuable but Safe Starting Point
Our surveys also reveal that getting started is a barrier. One way to cope with this barrier is to identify a preexisting program or solution and integrate big data and/or Hadoop into that program. That way you have a manageable, incremental update that can add more value to an already valuable solution. As a bonus, this approach defines a business case that will help you get approval and support for your new work with big data and Hadoop.
In my world I regularly see specialists in data warehousing, data integration, reporting, and analytics integrating new big data into older reports, analytics applications, and analytics data sets. Similarly, they are integrating Hadoop to be one of many data platforms in the multiplatform environments that data warehouses have evolved into. Based on these users’ success, I know that the incremental approach is a safe and effective starting point.
Once you have skills and experience with big data and Hadoop, you can move on to creating new solutions. For example, many organizations want to expand their programs for advanced analytics to leverage new big data and also to run the business on analytics insight. For these programs, big data is important source material (along with traditional enterprise data), and Hadoop can be a both a storage strategy and an analytics processing engine.
Further Reading
Want to learn more about big data, Hadoop, and related subjects? The following TDWI Best Practices Reports are available via free download:
Hadoop for the Enterprise (2015)
Integrating Hadoop into Business Intelligence and Data Warehousing (2013)
About the Author
Philip Russom is director of TDWI Research for data management and oversees many of TDWI’s research-oriented publications, services, and events. He is a well-known figure in data warehousing and business intelligence, having published over 600 research reports, magazine articles, opinion columns, speeches, Webinars, and more. Before joining TDWI in 2005, Russom was an industry analyst covering BI at Forrester Research and Giga Information Group. He also ran his own business as an independent industry analyst and BI consultant and was a contributing editor with leading IT magazines. Before that, Russom worked in technical and marketing positions for various database vendors. You can reach him at [email protected], @prussom on Twitter, and on LinkedIn at linkedin.com/in/philiprussom.