LESSON - Better Business Intelligence for New Competitive Environments: How to Move beyond Static Data and Exploit Real-Time Resources
By Rick Kawamura, Director, Marketing, Kapow Technologies
Businesses that quickly discover valuable data gain significant advantages. First-movers get to market faster. First-responders reduce the impact of negative events. First-reactors capitalize on opportunities before anyone else.
The key, however, is to act on the right information. Timeliness, quality, and understanding are crucial. New technologies can help.
Conventional Methods Obsolete
In the past, traditional business intelligence (BI) methods manually gathered “structured data” from various public and private sources. The information would be assembled in spreadsheets and other decentralized tools via slow research and cut/paste methods. These approaches consistently fail, however.
Why? The information is stale, gamed, or incomplete—or all of these combined. You may be responding to obsolete non-issues, for example. Your limited view of market, financial, and operational data may be downright delusional in some cases.
“Web Speed” Changes Everything
Unstructured data is different. This data—the kind that’s constantly accumulating on the Web in e-mail inboxes, corporate knowledge management systems, partner Web sites, Web applications, and social media sites—accumulates much faster than traditional BI data. Unstructured data could also include data siloed away in internal procurement, shipping, and CRM systems.
Until now, traditional BI systems have been unable to use unstructured data. It accumulates rapidly and is difficult to harness and make sense of. Unstructured data includes:
- Free-form Web content
- Partner and supplier Web sites
- SaaS and cloud applications
- Documents and PDFs
- E-mail
- Flash and JavaScript
- Password-protected portals on supplier sites
- Images
- Video and audio files
- User-generated content
- Secured access applications
- Social networks/social media
If you fail to capture this information, you’re not seeing an accurate picture of reality. Previously, you might have relied solely on monthly or quarterly government job reports, for example. With unstructured data, you can monitor real-time job listings on local or national job sites. The same applies to economic reports. Today you can build accurate reports for your city or ZIP code. For example, you might examine whether local companies are hiring sales people (an indication of growth) versus R&D (re-trenching for the future).
Organizations need to “radically improve the speed, accuracy, and cost efficiency of collecting and integrating the right data—from the Web and other data sources—at precisely the second they need it.” —IDC
What Are You Missing?
All kinds of companies are missing data trends that immediately impact their competitive position. The information comes from various sources, including:
- Public Web: Think real-time competitive intelligence, competitor price monitoring, customer sentiment, potential service issues, buzz monitoring, financial research, investment analysis, teen trends, and antipiracy.
- Internal systems: CRM, HR, and purchasing and finance systems can provide automated, real-time updates to your data analysis and decision making.
- External applications: Partner applications, supplier pricing, bank balances and transactions, procurement cycles, inventory tracking, supply chain, regulatory compliance, data syndication, online account aggregation, news syndication, and background and credit checks.
- Public opinion and perception: Social media and networking channels constantly pump out self-published, non-vetted public sentiment and feedback.
- Open source intelligence (OSINT): Monitoring communities, media, social networking, video sharing, wikis, and blogs to enable the intelligence community to fight terrorism or identify pandemics on a global scale.
This kind of analysis is not just for Fortune 500 businesses.
Requirements and Best Practices
Ideally, you want to set up automated processes to gather unstructured data. Kapow’s data collection, transformation, and integration solutions quickly find, clean, and feed unstructured information into business applications—in essence, a large-scale data integration method for the real-time enterprise.
Before that happens, however, you need to thoroughly understand the different types of information available, and then identify data that matches your particular goals and needs. Accessing the data is important, but you also need to interpret or transform it into meaningful information (i.e., add structure).
Some additional considerations:
- APIs are a hassle. Most Web data is not accessible via APIs, and even if you had access you’d have to custom code to access the information. A realistic solution needs to look elsewhere for data capture.
- Both the unstructured data desired and the tools and processes receiving the information require thoughtful consideration. You need to examine them closely to make it work.
- The reliability of captured data needs to be scrutinized. More often than not, it originated outside of your organization’s control.
- Gathering and visualizing unstructured data can place an enormous burden on IT.
This last point is particularly relevant. Integrating unstructured data into BI platforms is a highly complex process and can pull IT staff from other priorities. Transforming unstructured data into dashboard-usable information can burn lots of time if done manually. You need an automated or semiautomatic intelligent system.
A Race for First
Companies are now positioning to be first. They want to be first to identify or capitalize on opportunities using all the available information. Those that figure out how to capture real-time data streams and convert that into relevant, meaningful, actionable information will win.
This article originally appeared in the issue of .