TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

The Maturation of Hadoop -- Adoption and Experience

New survey results suggest that even when organizations have decided to commit to some use of Hadoop, their teams may still need additional training to come up to speed.

• By David Loshin
• September 30, 2016

From the volume of technology media articles about Hadoop and its evolving ecosystem, it seems that the adoption of big data processing and analytics is pervasive across almost all industries.

This perception was the motivation for our recent research survey at Decisionworx: "Hadoop in Transition." We reached out to a small but focused group of data management and application development professionals and posed a series of questions about their experiences evaluating, testing, and adopting Hadoop-based technologies within their organizations.

Responses were analyzed by segmenting the answers based on duration of corporate experience with Hadoop and assessing differences about product choice, vendor satisfaction, and level of adoption and implementation maturity.

Survey Says: Plenty of Interest, Not Much Experience

Our interpretation of the survey results suggests that although some organizations may have decided to commit to some use of Hadoop, their teams may still need some additional training to come up to speed. In particular, consider these findings:

There is a big appetite for knowledge about Hadoop. Interestingly, among the respondents indicating that they worked in IT, many reported that their organization had no current plans to use Hadoop. The fact that these individuals were participating in the survey may indicate a growing interest in what the Hadoop ecosystem has to offer and a desire for awareness as opportunities for adoption emerge over time.

Organizations are still in the early stages of Hadoop adoption. Among respondents, 52 percent worked for organizations that had less than two years of experience, 30 percent worked for organizations with between two and four years of experience, and approximately 18 percent worked for organizations with four or more years of experience.

Many are at the early consideration, evaluation, and casual-use stages of adoption, which may be a good sign. As the Hadoop ecosystem matures, vendors may take the opportunity to engage new and recent Hadoop converts -- helping them with application design, development, and integration within the enterprise production environment.

Individuals need to hone their skills. When we looked only at individuals from organizations that have been working with Hadoop for more than two years, almost 23 percent of respondents indicated they did not yet personally work with Hadoop, and 29 percent had less than two years of experience.

When looking at the responses from those at organizations with less than two years of Hadoop experience, over 54 percent of respondents said that they did not yet work with Hadoop. Organizational Hadoop adoption is still in its early stages, and it appears that the pool of skilled resources remains correspondingly limited.

Bridging the Skills Gap May Be Key to Success

Our conclusion is that one potential barrier to Hadoop success is bridging the skills gap. Engaging experienced resources with Hadoop skills at an early point in the adoption and production stages is potentially very difficult.

For an organization to remain competitive, it may be valuable to proactively establish Hadoop education and training programs. That way, current staff members can come up to speed and can track with the technology as it is brought in-house.

Although there are a number of reported Hadoop success stories, many organizations remain at an early stage of Hadoop adoption. The Hadoop environment and its related components have developed at a fast pace, but one byproduct of the rapid introduction of the technology stack is this apparent gap between years of corporate experience and personal exposure to the components of the Hadoop ecosystem.