How to Guarantee Big ROI on Big Data
You can't process big data with small data technology.
[Editor's note: This article was originally published at Data Science Central and is reprinted by permission of the author.]
By Vincent Granville, Data Science Central
I will start this article with an analogy:
- Processing big data with small data technology is like building an 80-story skyscraper brick by brick. It will take a lot of time just to get to floor #10, and it will collapse by the time you reach floor #15.
- If you build a 80-story skyscraper without good ROI analysis, maybe you will end up with a building that has lots of unoccupied space, generating a negative ROI. The same applies to big data. You must do due diligence before embarking on big data projects.
To put it differently, do you think it is worth spending $10,000,000 on a very expensive weather prediction system that is supposed to provide perfect predictions every 5 minutes, anywhere in US? After all, my own prediction system "tomorrow will be the same as today" works just fine for me (I live in Seattle - and this prediction beats many advanced algorithms). But recently, with my wife complaining that we can't schedule a nice dinner on a terrace or a hike in the mountains until the very last minute (when restaurants are full booked), I am wondering if using a better prediction tool could make sense, even if it means spending money. And what about airline companies interested in reducing bumpy air and detecting micro-bursts to increase client satisfaction, diminish aircraft wear, and reduce plane crashes? How much great weather predictions are worth to airline companies, the military, or hotels?
One of the main reasons big data ROI seems so obscure to CEO's is because CEO's don't use the right people to assess ROI on big data - they might indeed use nobody, but gut feelings instead. There is still a lot of confusion about what a data scientist is, and I believe this is one of the major bottlenecks against adopting big data. Executive people erroneously think that hiring a data scientist trained with a respected university degree is the solution, but many times it fails because they are hiring a fake data scientist. Professionals with very advanced statistical knowledge won't help you much. You need a guy like me, who maybe is not as statistically or computer science savvy as my peers, yet has incredible business experience and both broad and deep business knowledge - including law, human resources, operations, product, accounting, sales, client relationships and analytics. In short, a vertical data scientist. Such a person is very hard to find. In my case, I would ask a salary well above $500K per year, making it a bad hire for several reasons (not just the salary). I would not accept a data science position in any company big or small, except with companies where I am the founder.
So what is the solution? Working with a guy like me for about 20 hours to help you jump-start your big data projects and assess expected ROI, in a role very similar to a management consultant.This is the real solution to the problem.
I can give you a few examples where I successfully helped a company benefit from big data. Let me mention one example that really shows the challenges that companies are facing with big data. This is one example - our company - where we leveraged data to the fullest possible extend, with great ROI. It started as follows: paying a significant amount of money to a newsletter management vendor each month, and getting very detailed analytics about our email campaigns (clicks, opens etc. broken down per link, segment and per campaign). This highly granular information, while valuable, had limited potential. One day though, I realized that Gmail - our top segment - sometimes had a very poor performance. We analyzed the data, tested with other vendors, fixed the problem, and never experienced the problem of 1.5% open rate for Gmail (Gmail represents 30% of our subscribers and growing steadily, and by far the best subscribers). Since the problem was detected and fixed, we are consistently above 20% open rate. The return, in terms of satisfied clients, is tremendous for us - to the point that we are killing all our competitors.
So how did this happen? Data from our vendor is not broken down by ISP. But it is granular enough that we can break it down ourselves. The big question is: how did we ever thought about doing a break-down by ISP? This is where having a creative data expert, with immense domain expertise (rather than immense theoretical knowledge) helps. An hiring mistake would be to try to hire a guy like me, give him a chief data scientist job title, and pay him $500k/year. This won't work. But hiring a guy like me in a management consulting role, maybe a $10,000 project, is the way to go. By the way, I'm full booked for the whole year, interestingly on projects that do not involve consulting. So you would have to find someone else.
For those vendors selling analytics, I believe it would be very useful if you sell analytics as part of a bigger offering. For instance, in our case, we spend $20K per year in newsletter management services. It comes with analytics, but it is sold as a newsletter service, not as an analytic service.
What you need is a guy who is both a data hacker, a data scientist, and a management consultant, a couple of hours per month, to detect and turn data into gold, and recommend solutions after a financial analysis. If you can't do it, your competitor or bad hackers will do, at your expense.