Big Data: Benefits, Challenges, and Best Practices
If big data is such a popular technology, why haven't more enterprises adopted it? What benefits can you expect, and what best practices can make your big data project a success?
- By James E. Powell
- June 26, 2012
If big data is such a popular technology, why haven't more enterprises adopted it? What benefits can BI professionals expect, and what best practices can make your big data project a success? Where will big data be in five years?
For answers, we turned to Krish Krishnan, CEO of Sixth Sense Advisors, Inc. and a recognized big data expert. Krishnan is co-author (will Bill Inmon) of Building the Unstructured Data Warehouse and is teaching classes on big data subjects on August 3, 2012 at the TDWI World Conference in San Diego.
BI This Week: What is the business value of big data?
Krish Krishnan: Big data provides business insights beyond traditional data from transactional systems. These insights offer valuable perspectives on human behavior, sentiments, interactions, and utility usage trends from a business-to-consumer model, and insights into contracts, compliance, and legal and financial trends from a business-to-business perspective. Every organization can benefit from expanding their analytics beyond structured transactional data by integrating both big data and data warehouse (DW) analytics, as one can perform deeper contextual analytics that was not possible till a few years recently.
What are the advantages business intelligence (BI) professionals can expect from big data analytics?
Big data provides access to data that was always available but not consumable. The expansion of big data into business intelligence platforms enables BI professionals to leverage expanded analytics and create 360° perspectives. Let me provide a few sample areas that can benefit from BI and big data analytics.
For customer relationship management systems, you can create powerful visualizations of customer sentiments, wish lists, and actual customer response data from campaigns to measure true campaign effectiveness. You can model and predict customer behaviors by integrating data across call centers, blogs, forums, and social media platforms into deeper analytics. You can deploy better call-center metrics for customer management or even create an effective micro-targeting and micro-segmentation model for new customer acquisition with better response rates of acceptance.
If your enterprise deals with products and/or services, with big data analytics you can create powerful models for trends, behaviors, and markets, and you can solve research and ideation issues by leveraging "crowd-sourcing" models and embedding analytical results from your work. If you work in the utility industry, you can create predictive models of consumer markets by deploying technologies such as a smart grid. This would create revenue opportunities in advisory services and provide better models for rate management.
Another popular area where big data analytics is being used is health care. For example, service providers can leverage big data to deploy Body Area Networks, helping lower patient costs while providing "patient-centric" services. Lowering costs and enabling efficiencies are critical goals for hospitals, nursing homes, and caregivers. Another application of big data is to optimize clinical trials to prevent errors, reduce costs, and ensure compliance and ensure you're meeting regulatory requirements consistently. Although these analytics are partially fulfilled today, their expansion will enable proactive approaches rather than reactive ones.
There's considerable focus these days on new technology -- and it's possible to integrate survey, social media feedback, and participation information into traditional platforms. These data points can be represented in analytic and reporting visualizations, helping hospitals and care providers (for example) to improve their quality of service
What is different about big data analytics compared to traditional analytics?
Traditional analytics are developed and deployed based on structured data. The problems solved by analytical models developed on structured data provide insights but often fall short in predictive and indicative analytics. The critical reason for this failure is the lack of near-real-time information and expanded information beyond structured data. This is where big data analytics enables better analytical insights -- by integrating more voluminous data of varying complexity and timeliness into one structured output.
After integrating text, voice, streaming data, and unstructured data analytics into one structure or model, we can harness the different views of related information into analytical models. These models can generate powerful, multi-dimensional metrics that can be leveraged with traditional analytics.
Why aren't more enterprises adopting big data analytics? What are the top two or three impediments?
Adoption of big data is slower than expected in traditional enterprises. There are several reasons for this. At the top of my list is that current business models and goals don't require big data integration. Furthermore, there is no perceived additional value offered by big data as to the organization -- there is no clear business case articulated, and thus no business value calculated.
There are other inhibiting factors. Executives lack an understanding of (and thus sponsorship of) big data, which also brings processing complexities that create additional stress on IT teams (in terms of maintenance) and business teams (in terms of adoption and usage). In these times of tight budgets, IT teams simply do not have the necessary bandwidth to implement yet another new system or technology.
What best practices can you recommend business teams to follow to ensure project approval?
Business teams must own the big data project or program. This requires the CMO, CSO, CRO, CFO, or CCO to become the executive sponsor of the initiative. To gain such sponsorship, teams need to build a robust business case, providing the depth of data analysis required, why the enterprises lacks such depth in the current environment, as well as the key metrics and analytics that can be implemented or extended with a big data project.
The business case must also provide a clear ROI period and repeated ROI cycles in the system life cycle. A clear and concise business case will be the first step towards getting project approval.
Once a big data analytics project is approved, what are the biggest problems facing an enterprise?
Big data projects are complex to plan and execute. The complexity stems from your need to perform data discovery before you can document a single user requirement. If you lack clear business requirements, you cannot plan the remaining project logistics, including team, skills, execution steps, rollout, and training.
Another issue with big data projects is the lack of availability of business subject matter experts (SMEs) who have both data and institutional knowledge as well as a command of math and logic. Organizations often shy away from big data due to this critical resource -- the data scientist.
What best practices can you recommend to avoid these problems?
In order to implement a big data project, here are a few tips.
First, create a powerful team that can set up a platform to explore big data. This team will work with business and data analysts to create the road map for further execution. Critical success factors include:
- Availability of IT resources to build and configure the selected big data platform
- Availability of business SMEs with data and domain expertise
- Availability of resources with BI expertise and deep statistical knowledge
- Implement a technology center of excellence to provide big data infrastructure support
- Extend other BI best practices, including data governance, MDM, metadata management, analytics definition, and visualization, to include big data
- Ensure adequate training for users to understand the new data and its integration into the analytical and reporting platforms
When it comes to people, having a combination of individuals mentioned above will create a team that can leverage each other's skills and create a unified vision for exploring big data.
Where is big data headed? Where do you see the technology in, say, the next year or two?
Big data is headed the same path that business intelligence and analytics did a decade ago -- to become the crown jewel of enterprise data management. It will be the most important project to be executed by any enterprise in the next two to five years. This growth and adoption of big data will be aided by continued technology growth in Hadoop, NoSQL, MapReduce, and Mahout algorithms, among others.
I definitely see an evolution of in-memory processing and cloud computing to be the infrastructure for executing the current technologies in the future. There are also efforts underway to create hybrid data platforms to manage big and structured data on one engine from companies such as Teradata, EMC, and Hadapt.
The journey is long and complex, but with guided navigation from concept to adoption, big data will continue the dominance into the foreseeable future.