View online: tdwi.org/flashpoint
July 10, 2014
ANNOUNCEMENTS
NEW TDWI Best Practices Report
Business-Driven BI and Analytics
NEW TDWI Checklist Report
Using Streaming Analytics for Continuous Operational Intelligence
NEW TDWI Checklist Report
Adopting Next-Generation Data Technologies
NEW TDWI E-Book
In-Memory Technologies and the Modern Data Warehouse
NEW TDWI E-Book
Why Enterprises Need Trustworthy Data
CONTENTS
Why BI Projects Fail: Data-Driven Data Model Design
Bridging the Analytics Skill Gap with Crowdsourcing
Infrastructure for Predictive Analytics
Mistake: Underestimating the Data
See what's
current in TDWI Education, Events, Webinars,
and Marketplace
Why BI Projects Fail: Data-Driven Data Model Design
Steve Dine
Datasource Consulting, LLC
Topics:
Data Analysis and Design
Many people look to agile development methodologies as a structural way to involve the business in the design and development of business intelligence (BI) solutions. However, many times the design practices employed within the iterations aren’t actually business driven at all. We see this most often in the design of the data models and BI semantic layers, where data modelers take high-level user requirements and design the model based on their analysis of the data. A compelling alternative is “business driven” data model development, whereby the modeling process starts with the users, not the data. How is business-driven development different from its traditional counterpart, and how--if at all--is it better?
Generally speaking, both paradigms involve the business. In the data-driven model, however, the business's role in data model design is severely limited. The stakeholders generate their list of requirements, sometimes in the form of user stories, and the data modeler begins designing the data model, often creating a design based on how the data is modeled in the source systems, rather than how it supports the business processes. When the users begin working with the data, they soon identify new, specific requirements and/or business rules.
Because of its structure, a system-driven approach is highly susceptible to failures of communication and accuracy. Depending on an organization's culture or maturity level, these endemic problems can complicate--and sometimes sabotage--development. The system-driven model also has an absolute-latency feedback loop: potential issues are identified (or fixed) late in the project, which has a significant impact on downstream development. The result is that a data modeler must continually adjust the model as new rules and requirements are identified. Besides producing suboptimal data models, this disrupts ETL and analytics development, causing projects to miss their target delivery date. In a business-driven approach, by contrast, the data modeler avoids making significant changes to the model because more requirements and business rules are fleshed out up front. Business-driven development addresses these and other issues.
Fit and Finish
With business-driven modeling, the initiative begins with and is driven by the business. Business subject matter experts work with the data modeler to refine and clarify their requirements via conceptual and logical data modeling, providing the opportunity to validate the business process and the business rules and nomenclature.
For this process to work, both business users and data modeler must understand the business in the same way. This is why business-driven development places so much emphasis on conceptual and/or logical modeling. A conceptual model is a high-level representation of the business process. However, in the real world, there isn't always enough time to perfect a conceptual model, so some may decide to start with a logical model, which can be abstracted to a higher level. What's important is that both business and IT see and understand the business, its key concepts, entities, rules, and relationships in the same way.
A business-driven approach promotes near-real-time feedback and correction and makes it possible for the business to more quickly identify what it actually needs versus what it thinks it wants. The data modeler plays a key role by facilitating discussion with business SMEs to flesh out concepts, entities, rules, and relationships. In a sense, business-driven development is dialectical: it's asking and answering questions, which leads to new questions and answers, along a discussion thread. In practice, this dialectical process helps mitigate common problems, such as what happens when the data modeler just misses something because of misunderstanding or imprecision.
Better Project Outcomes, Better Business/IT Relations
Business-driven data modeling is an effective strategy for aligning the perspectives and priorities of business and IT, ensuring that both share an understanding of the business. In business-driven data model design, stakeholders and subject matter experts work closely with the data modeler to identify and refine their analytic requirements. This collaborative process helps both groups clarify business rules and nomenclature and gain insight into business processes and operations. It involves asking and answering questions, which leads to new questions and answers. The result: business users can develop a better understanding of what’s actually needed and identify the potential value of the business problem they are trying to solve.
Most important, business-driven development leads to empirically better project outcomes. Problems can be proactively identified and corrected prior to development and testing, rather than late in the process. Finally, business-driven data modeling can be as valuable a tool for business self-awareness as for IT-business alignment. By working with the BI group and data modeler, the business interrogates itself and its operations. Stakeholders discover how their business functions and where it excels, and where they might focus on improving operations. Business users receive better, usable data that addresses their actual needs--and they get it faster, building trust and respect between both groups and promoting more effective collaboration in future endeavors.
Steve Dine is the managing partner and founder of Datasource Consulting, LLC. He has extensive experience delivering and managing successful, highly scalable, and maintainable data integration and business intelligence solutions. Steve is a faculty member at TDWI and a judge for the annual TDWI Best Practices Awards.
Bridging the Analytics Skill Gap with Crowdsourcing
Ryan Hart and Troy Hiltbrand
Few areas in the economy have generated as much attention recently as big data and advanced analytics, which have the potential to revolutionize the way businesses function in the coming years. However, organizations seeking to implement the advanced analytics that can provide a competitive advantage face a major challenge: finding the elusive data scientists needed to execute on big data strategy. This article looks at how some businesses are bridging the gap between vision and reality.
Learn more: Read this article by downloading the Business Intelligence Journal, Vol. 19, No. 2
Highlight of key findings from TDWI's wide variety of research
Infrastructure for Predictive Analytics
Most of the respondents using predictive analytics today have a tried-and-true infrastructure in place for it, such as a data warehouse (89%), desktop applications (78%), or flat files on servers (75%). However, newer components are finding their way into the mix.
Analytic platforms and appliances are gaining in popularity. Although the active group is currently running analytics using the tried-and-true infrastructure, there is definitely a move to newer forms of technology to support predictive analytics such as the analytics platform (55%) and the appliance (41%). Analytics platforms can range from a database management system that manages data for analytics (typically query based), to a tool featuring a collection of algorithms, to any platform that happens to have analytics embedded. In addition, Figure 14 shows that if respondents stick to their plans, more than 80% of the active-group respondents will be using analytics platforms in the next three years. Slightly more than 60% will use an appliance of some kind.
Likewise, those investigating the technology (not shown) are also looking at analytics platforms (73%) and appliances (46%) over the next three years to help support their predictive analytics needs.
(Click for larger image)
Read the full report: Download Predictive Analytics for Business Advantage (Q1 2014)
FlashPoint Rx prescribes a "Mistake to Avoid" for business intelligence and data warehousing professionals.
Mistake: Underestimating the Data
Fern Halper
It’s one thing to dump lots of data into Hadoop or store it in other data repositories. However, getting the data in shape to analyze without understanding how complex it can be is another matter entirely. Data quality was discussed in Mistake Four. Additionally, in TDWI surveys on big data challenges, data integration often ranks at the top of the list. Yet some companies underestimate it. Integrating big data for analysis is complex for several reasons: the data is often siloed; the data can come from multiple sources with no metadata or master data; and the data can be multistructured. Couple that with the possibility that some of it might be real-time data, and you can see how complex big data can be.
Big data also introduces new types of data. Twitter streams, Facebook posts, sensor data, security logs, video, and other sources of data are emerging daily. However, in order to ultimately take action, you need to analyze the data and introduce it into the operating processes of your company. Smart organizations often put together a strategy with a road map for dealing with new data sources and modifying existing workflows to accommodate big data or create a new big data workflow, if the data source will be used in an ongoing way.
Read the full issue: Download Ten Mistakes to Avoid in Big Data Analytics Projects (Q2 2014)
