TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

We've Already Failed at Big Data -- Now What?

If your team has been taking "too long" to finish its big data project, here are five steps to take now to regain trust and respect and show the business benefits of your work.

• By Jill Dyché
• September 20, 2016

The desire to adopt the latest emerging trend burns hottest in the high-tech sector. A few years ago, as big data was heating up, executives and technophiles rushed to acquire big data solutions faster than you could say "Wikipedia."

As with all hot trends, however, big data might be losing some steam. Wearables, artificial intelligence, virtual reality, and biometrics are all undergoing constant revision and tweaking. Big data is no different. Read Naveen's question below as a cautionary tale:

Dear Jill,

I appreciate your column and my team and I thought we would ask your opinion.

I work for a specialty retailer. Two years ago, as part of a larger omnichannel program handed down from our board, we launched a big data initiative. We secured a Hadoop distribution, signed on to several open source projects, modernized our analytics, and created a big data team to start loading data.

As the project manager, I can tell you that we have all worked very diligently and for many hours to make our big data platform work. My teammates and I started the data warehouse project here 15 years ago. We have learned a lot since then, and we all consider ourselves big data experts.

However, things have slowed down. We continue to load different types of data (including point of sale, customer profile, Web logs, and supplier data) and we've actually discussed using the platform to tie together data that had never been integrated.

I'm hearing some hallway chatter about the project being "in the weeds." I think my colleagues are beginning to lose faith in me and my team. What should we have done? More important, what should we do now?

-- Naveen from Ohio

Well, to say we all saw this coming -- not with you, Naveen, but as part of the inevitable backlash -- would be to rub salt in the proverbial wound. In a 2012 Harvard Business Review article, McKinsey researchers Dominic Barton and David Court compared big data to customer relationship management efforts a dozen years earlier:

Too many C-suites were blind to the practical implications of new CRM technologies -- namely, that to capitalize on them, organizations would have to make complex process changes and build employees' skills. The promised gains ... were often slow in coming because the systems remained stubbornly disconnected from how companies and frontline managers made decisions, and new demands for data management added complexity to operations.

Substitute CRM with big data in that paragraph. The parallels are unmistakable.

Although comparing big data to the beleaguered tech projects of yore might seem trite, so are the solutions. Just because big data is new doesn't mean its guiding principles have to be.

I have to ask: Didn't you and your data warehouse team learn that data loading is a means to an end? Truth is, no one is really all that interested in data loading except the data loaders. Business people don't care about the data's journey as much as they care about its destination. They want to know that the data is authoritative so they can be comfortable making decisions with it.

Enough finger wagging. Here's what I would do if I were you:

1. Identify a strategic need. Maybe your company is thinking about sending personalized digital coupons via SMS messages to customers in the vicinity of a store. Maybe there's a supply chain optimization project that could use streaming data from "smart shelves" to detect inventory levels. Perhaps provisioning video data from stores to experiment with product placement might help. These are real-life examples from other retailers and you'll have some, too.

2. Establish a small, controlled project. Now that you've zeroed in on a need, identify a slice of new functionality where big data can help, and deliver it.

3. Apply rigor and talk about it. Now is the time to formalize big data development processes. (Hint: They are different from data warehouse development processes.) Be careful not to make this a science project, but at the same time show that your team has optimized the way it implements big data platforms and applications. Explain how that's agile. This will help you explain what's taken so long.

4. Tell a story. Usually the story goes something like this: "Before we added big data to [Project a] there were problems with [Issue b] and business people couldn't [Action c]. Now that [data area(s)] have been delivered, we've shown a significant improvement in [Benefit z], which has in turn delivered [cost savings/revenue generation/compliance results/customer loyalty increase description]." Seriously, before-and-after benefit scenarios are tragically rare in business so executives pay attention to them. This might even help resuscitate interest in big data.

5. Know what's next. Even before your first success, it's wise to gear up for your next activity. Institutional memory is short and your honeymoon is over. Gauge corporate priorities, available budgets, and skills, then share your delivery pipeline and solicit feedback.

This is your way out, Naveen. Too many teams bask in the glow of the new without realizing that no one wants to invest in an academic effort devoid of business value. This is true for the technology trends mentioned above and it's true for big data. It's not just the technology that's at stake here but your team's reputation.

No pressure. Good luck!

Now it's your turn! Email me a question about analytics programs, data management, organizational issues, or culture at [email protected].