Evolving Your Developers into Next-Generation Big Data Experts
How should organizations go about staffing up for big data?
By Nobby Akiha, Senior Vice President of Marketing, Actuate Corporation
How should your IT team prepare for big data?
Without a doubt, your current IT team is technically capable of taking on big data, having strong backgrounds in varieties of networks, databases, programming and understanding distributed systems and so on. However, they probably don't have the experience yet of how to tackle big data.
To get results with your big data project, you need to collect such large volumes of data that you need to either filter it to death to find the small data that's interesting, or move your lens so far back that you can examine the data as a whole. This can be a challenge as old-school BI programmers are not used to doing this.
However, there's no reason why they can't acquire the right skills -- with the right help. You can assist them by creating a culture where questions are asked in new ways. For example: asking how many network pings were made from a server in a day doesn't lead to any useful business insight. Instead, you need to start asking open-ended and qualitative questions, such as"What does this data show about where that business unit is heading?" or "What would success look like if we launched a product into this space?"
After all, big data is about finding those pieces of data that produce game-changing insights. Such realizations may be the result of a varietyof data types: cause and effect, related or correlative, branched, or data with dependencies. More often than not, that data will come from sources that are unlike the stats and facts we're used to working with in traditional BI.
Big data also comes from sources that can help predict what the customer will do next, or what the organization needs to do to minimize expenses and maximize efficiency. This means that although most developers are happy traversing database trees, now they have to work with a more distributed processing system that may or may not have the same structure, or may in fact have no structure. The result set from a big data probe doesn't have to look anything like the structure of the original query or the data model that you're working with; all it must do is conform to the shape and size of the answers the business is looking for. This is precisely why it's so important to know the questions before you start, of course.
You need to start teaching these new ways of looking at the data as part of the IT team's core disciplines. This new way of questioning is not the only "new trick" they need to learn; they also need to become a coach to the business, learning to teach executives how to ask the right kinds of questions, before helping them gather the data they require. In turn, your new-style big data developers need to understand how to map that line of inquiry back to the data sources to find the answer to a complex business problem.
The reality is many developers don't have the experience or desire for handholding the business like this. You may need to invest in soft skills training, such as how to elicit the right questions to ask and how to recognize what a good answer looks like (hint: it's not a SQL schema!). The developer's goal in each and every one of these conversations should be to come up with questions that are answerable with the available data sources or by stringing together a series of different yet related and sequential questions.
Welcome to big data and The Next Generation IT Team. It's a lot more like treasure hunting and detective work than building Oracle tables; but it's also a lot more fun.
Three Tips for Your Big Data Journey
Finally, in terms of preparing your IT staff for their evolution to the big data ways of questioning and thinking, remember the following.
First, you're not running a big data project, you're running a big data business process. Remember, analytics is a process because its requirements change all the time thanks to the nature of analytics: it continuously generates new questions, which inevitably lead to a need for more data sources. Big data analytics is no different, just bigger.
Second, this is a journey, and your IT team's knowledge and experience will be built up over time as the process unfolds, not all at once before the process begins. To be clear, your developer team members don't need to be data scientists (though familiarity with statistics is helpful). They don't need to be computer linguists, as natural language processing technologies will evolve and become easier, and they do not need to be expert MapReduce programmers or Hadoop experts, at least not yet. The data science, linguistics, and programming skills will all follow in time.
Finally, you want your developers ultimately to become true experts in understanding how to solve the business problems your organization faces -- as well as curious about some yet-to-be defined problems that will push their thirst for more data. The better they become, the more competitive your organization will be.
Your big data journey is inevitable. The sooner you begin, the sooner you will see success. Big data is your future. Make it a successful one.
Nobby Akiha is senior vice president of marketing at Actuate, the makers of BIRT Analytics and the BIRT-based suite of commercial products for development and deployment of custom business analytics applications for big data, predictive analytics and customer communication management. You can follow Nobby and Actuate at @Actu.