TDWI FlashPoint Newsletter TDWI FlashPoint Newsletter
Sponsored by:


TDWI Data Warehousing Concepts and Principles: An Introduction to the Field of Data Warehousing

The Future of Data Warehousing

TDWI Business Intelligence Fundamentals: From Data Warehousing to Business Impact


TDWI Checklist Report: Seven Steps to Implementing Data Discovery

Realize Unparalleled Business Value from Enterprise Data
Data Quality White 

Paper Data Replication for Business Intelligence and Data Warehousing
Presented by Philip Russom
Event date: October 13, 2011

Data Quality White 
Paper Business Intelligence Competency Center: Best Practices to Deliver REAL Business Value
Presented by Claudia Imhoff
Event date: October 18, 2011

TDWI Experts is a twice-monthly e-newsletter where BI/DW thought leaders share opinions and commentary about relevant industry topics and the latest technologies.

Article Image

October 6, 2011

Text and Sentiment Analysis: The Customer's Voice is Heard

David Stodder
Director of TDWI Research for Business Intelligence

Topic: Analytics

Sentiment analysis is hot. The strong interest is driving the emergence of text analytics – the underlying technology and method for sentiment analysis – into the mainstream. Organizations want to know what customers, reviewers, competitors, and other "voices" are saying about their products, services, and brands. They want to discover the leading sentiment influencers and their networks, track what is expressed after major events (such as product launches) and bring the feedback into their marketing and product management.

They also want to use the insights as context for more traditional business intelligence and data analytics that are focused on measuring business performance and evaluating customer profitability.

Text analytics software tools allow organizations to perform sentiment analysis at the speed and scale necessary to make the effort worthwhile. Diverse tools in the marketplace, primarily geared for use by experienced professionals, offer natural-language processing, relationship extraction, support for statistical and linguistic methods, predictive modeling, and more.

Many tools that used to call themselves text analytics products are now pumping up sentiment analysis as their category. Just as with BI and data analytics, a big challenge with text analytics is translating the findings into information that is accurate, understandable, and actionable for non-technical personnel.

Big companies with diverse product portfolios and millions of customers worldwide can easily find themselves looking at many terabytes of unstructured or semi- structured content that they want to analyze. Even before the advent of social media, interpreting the massive volumes of text comments was beyond what humans could do with any kind of speed.

As a result, it has not been uncommon for "voice-of-the- customer" files and other customer satisfaction feedback forms to pile up for months before an enterprise is able to analyze them. Text analytics tools allow organizations to increase speed through automation, and apply scientific methods so they can refine their analysis of content over time and create repeatable modes of inquiry.

It's still early, but the enormity of the sentiment analysis task is a key driver behind tighter integration between text analytics and powerful, massively parallel databases and data file systems such as Hadoop. MapReduce and data compression technologies are also critical to working with this type of "big data."

As we look ahead into 2012, clearly one of the most significant emerging technology trends will be deeper integration between text analytics and underlying database systems, which will enable organizations to apply more computing power to sentiment analysis. Mergers, acquisitions, and recent product announcements from all the major database vendors certainly indicate their interest in furthering such integration.

Sponsored Links

Pushing the Customer Focus Outward

Sentiment analysis is helping to turn the attention of organizations outward, into the customer's realm, to gain a better understanding of customer satisfaction, loyalty, and other important qualities. Even in most customer-facing operations such as contact centers, the focus is surprisingly pointed inward. Organizations devote considerable financial resources to developing quality assurance programs to ensure that contact center agents are following scripts correctly and adhering to internal rules and performance standards.

Even measures such as first-call resolution are often interpreted internally as more about managing costs and time spent per customer. Agents are typically evaluated based on internal measures. Organizations will use surveys to monitor customer satisfaction, but when the numbers are added up and aggregated, the actual customer experience is often lost.

Sentiment analysis is enabling organizations to measure something they previously could not. As Peter Drucker famously said, "You can only manage what you can measure." Now, organizations can use text analysis to measure how sentiments expressed outside their walls indicate levels of customer satisfaction. They can also link sentiments to actual events (such as product introductions) rather than wait for historical trending reports that make it harder to see such connections. Sentiment analysis will help organizations evaluate as well whether their contact center agent assessments are actually guiding performance in ways that actually improve customer satisfaction.

With competitive advantages often fleeting in today's marketplace, organizations need to use sentiment analysis to gain a more timely understanding of customer satisfaction as well as insight into how influencers and communities are shaping customers' opinions. Most organizations cannot gain this understanding from measures used in customer-facing operations, which are typically focused on internal concerns, such as costs.

As sentiment analysis (and underlying text analytics tool implementation) expands as an emerging technology, it will be interesting to see how the insights begin to alter common understanding in organizations about the quality of the customer experience they are providing.

David Stodder is director of research for business intelligence at The Data Warehousing Institute (TDWI). David can be reached at

Vendor Q & A - Sybase

Answer provided by David Jonker, Director of Product Marketing

Q: How are enterprises handling the growing number of business analytics users and workloads?

A: With a lot of pain and suffering. The majority of enterprises continue to deploy their reporting and analytics on transactional, row-based databases. Those databases don't scale for business analytics. IT departments are doing their best to compensate by introducing multiple, mirrored data warehouses, many data marts, heavy use of pre-calculated cubes, data aggregation, and table indexes. When you stop to think about it, essentially all of these exist to overcome the limitation of row-based databases to scale the number of users and workloads.

However, all of these approaches add massive amounts of complexity for IT. Complexity is overwhelming the IT department and, in extreme cases, crippling BI users from leveraging business analytics the way they could. BI users are still being limited in what they can do -- denied the ability to run ad hoc queries, store enough data for analysis, or allow more users to benefit from business analytics.

The leaders, on the other hand, are using columnar databases for their analytics and reporting environments because they overcome the scalability issues associated with row-based databases and can dramatically simplify the IT environment. Enterprises are also beginning to wake up to the possibility of using analytics grids to further expand the amount of data analyzed. However, if enterprises are not careful, they could choose a solution that can analyze more data but not necessarily support more users.

Copyright 2011. TDWI. All rights reserved.

TDWI Membership TDWI Membership TDWI Membership This message has been sent to: [-EMAILADDR-]
The Data Warehousing Institute (TDWI) will periodically send you information via e-mail about related products and services. If you do not wish to receive these types of e-mails, use our preference page:[-EMAILADDR-]

To view our privacy policy, visit:

TDWI, 1201 Monster Road SW, Suite 250, Renton, WA 98057 TDWI Info