By Wayne W. Eckerson
With “analytics” a hot buzzword and the newest competitive differentiator in Corporate America, SAS is sitting in the catbird’s seat. With $3.1 billion in sales last year, SAS is by far the leader in analytics software and is prepared to leverage its position to compete head on with the big boys.
It was a clear after spending a day with SAS executives in Steamboat Springs, Colorado that SAS is thinking as big as the towering Rocky Mountains that surrounded our resort hotel. Specifically, SAS is focused on competing with IBM. That’s partly because IBM just acquired its nearest analytics competitor, SPSS, and is now heavily marketing its “Smart Analytics System” initiative, which enables customers to purchase a single integrated analytical solution consisting of hardware, software, tools, consulting services, and support via a single SKU and have it delivered in two weeks. (See “IBM Rediscovers Itself” August, 2009.)
To bulk up against Big Blue and its 4,000 consultants, SAS has teamed up with Accenture, who is also feeling the heat from IBM, but in consulting services. Together, SAS and Accenture will spend $50 million initially to develop new analytics products for six industries. For its part, Accenture sees analytics as a big growth area and partnering with the top analytics software vendor makes sense. And to beef up its corporate image, SAS is building a new 280,000 foot executive briefing center with 690 offices, two auditoriums, and a full café to impress analytics prospects.
SAS has never been known for its partnering ability, but the stakes are high and both SAS and Accenture are motivated by a common, fierce competitor. In the past two years, SAS has honed its partnering skills by investing serious time and money with Teradata, the fruits of which are starting to pay dividends in six-figure sales deals. And now, SAS is aggressively partnering with a slew of upstart database vendors to move analytics into the database, even IBM DB2. This is an interesting surround strategy for SAS: wherever IBM goes, it will find SAS already embedded in the customers database of choice. Not bad!
Other highlights from the SAS Analyst Day:
- SAS executives believe social media analysis is a big potential growth area for the company, leveraging its strength in analytics as well as its 2008 acquisition of Teragram, a leader in natural language processing and advanced linguistic technology. This is a natural market for SAS to dominate, although it faces a slew of nimble startups and established players
- SAS’ data integration products, which are now housed under the DataFlux subsidiary, were the fastest growing parts of its portfolio, with more than 50 percent growth in some markets. While most of the products are not new, SAS executives say the growth comes from the fact that DataFlux provides vendor-neutral data integration products so is not tied to the SAS installed base. Plus, companies are mining ever-growing volumes of data and are recognizing the need for clean, integrated data. “More companies view fact-based decision making as a key to success and are coming to recognize that clean, integrated data is critical for delivering insights,” says Jim Davis, senior vice president and chief marketing officer.
- SAS is no longer overlooking the small- and medium-sized business (SMB) market, ramping up hosted offerings for customer intelligence and other applications that smaller companies can access. In 2009, it acquired 228 new SMB customers.
- SAS has an active hosting business for analytics customers who don’t want to provision servers in their own environment. SAS’ on demand revenues grew 30% and number of customers increased 43% from 2008 to 2009. SAS is building a huge new hosting center so expect more on demand offerings to come from SAS in the months ahead.
Posted on 03/07/10 at 12:53 PM1 comments
One of my takeaways from last week’s BI Executive Summit in Las Vegas is that veteran BI directors are worried about the pace of change at the departmental level. More specifically, they are worried about how to support the business’ desire for new tools and data stores without undermining the data warehousing architecture and single version of truth they have worked so hard to deliver.
At the same time, many have recognized that the corporate BI team they manage has become a bottleneck. They know that if they don’t deliver solutions faster and reduce their project backlog, departments will circumvent them and develop renegade BI solutions that undermine the architectural integrity of the data warehousing environment.
The Wisdom of Letting Go
In terms of TDWI’s BI Maturity Model, these DW veterans have achieved adulthood (i.e. centralized development and EDW) and are on the cusp of landing in the Sage stage. However, to achieve true BI wisdom (i.e. Sage stage), they must do something that is both counterintuitive and terrifying: they must let go. They must empower departments and business units to build their own DW and BI solutions.
Entrusting departments to do the right thing is a terrifying prospect for most BI veterans. They fear that the departments will create islands of analytical information and undermine data consistency with which they have worked so hard to achieve. The thought of empowering departments makes them grip the proverbial BI steering wheel tighter. But asserting control at this stage usually backfires. The only option is to adopt a Zenlike attitude and let go.
Trust in Standards
I'm reminded of the advice that Yoda in the movie “Star Wars” provides his Jedi warriors-in-training: “Let go and trust the force.” But, in this case, DW veterans need to trust their standards. That is, the BI standards that they’ve developed in the BI Competency Center, including definitions for business objects (i.e. business entities and metrics), processes for managing BI projects, techniques for developing BI software, and processes and procedures for managing ETL jobs and handling errors, among other things.
Some DW veterans who have gone down this path add the caveat: “trust but verify.” Although educating and training departmental IT personnel about proper BI development is critical, it’s also important to create validation routines where possible to ensure business units conform to standards.
Engage Departmental Analysts
The cagiest veterans also recognize that the key to making distributed BI development work is to recruit key analysts in each department to serve on a BI Working Committee. The Working Committee defines DW and BI standards, technologies, and architectures and essentially drives the BI effort, reporting their recommendations to the BI Steering Committee comprised of business sponsors for approval. Engaging analysts who are most apt to create renegade BI systems ensures the DW serves their needs and helps ensure buy-in and support.
By adopting a Zen like approach to BI, veteran DW managers can eliminate project backlogs, ensure a high level of customer satisfaction, and achieve BI nirvana.
Posted on 02/28/10 at 6:03 PM0 comments
More than 150 BI Directors and BI Sponsors from small, medium, and large companies plus a dozen or so sponsors attended TDWI’s BI Executive Summit last week, a record turnout.
Here are a few of the things I learned:
- Veteran BI directors are worried about the pace of change at the departmental level. More specifically, they are worried about how to support the business’ desire for new tools and data stores without undermining the data warehousing architecture and single version of truth they have worked so hard to deliver.(See "Zen BI: The Wisdom of Letting Go.")
- Jill Dyche explained that corporate BI teams can either be Data Providers or Solutions Providers. That this is an option is a new concept for me. However, after some thought, I believe that unless BI teams help deliver solutions, the data they provision will be underutilized. Unless the BI team helps solve business problems by delivering business solutions, it can never be viewed as a strategic partner.
- Most BI teams have project backlogs and don’t have a great way to get in front of them. Self-service BI can help eliminate a lot of the onesy and twosey requests for custom reports. BI Porfolios and roadmaps can help prioritize deliverables but executives always override their own priorities. Many veteran BI managers are looking to push more development back into the departments as a way to accelerate projects.
- There is a lot of interest in predictive analytics, dashboards, and the cloud. Those were the top three vote getters to the question, “Which technologies will have the most impact on your BI program in three years?”
- Most of the case studies at the Summit described real-time data delivery environments, often coupled with analytics. GE Rails applied statistical models to real-time data to help customer service agents figure out the optimal repair facility to send railroad cars to get fixed; Linkshare captures and displays Web activity and commissions to external customers (publishers and advertisers), and Seattle Teachers’ Credit Union delivers real-time recommendations to customer service agents.
- There was a lot of interest in how to launch an analytics practice and Aldo Mancini provided some great tips from his experiences at Discover Financial Services. To get SAS analysts to start using the data warehouse as a way to accelerate model development, he had them help design the subject areas and variables that should go into it. Then, he taught them how to use SQL so they could transform data into their desired format.
We’re already gearing up for our next Summit which will be held August 16-18 in San Diego. Hope to see you there!
Posted on 02/28/10 at 6:06 PM0 comments
Everyone wants to move beyond reporting to deliver value-added insights through analytics. The problem is that few organizations know where to begin. Here is a ten-step guide for launching a vibrant analytics practice.
Launching the Practice
Step 1: Find an Analyst. You can’t do analytics without an analyst! Most companies have one or more analysts burrowed inside a department. Look for someone who is bright, curious, and understands key business processes inside and out. The analyst should like to work with numbers, have strong Excel, SQL, OLAP, and database skills, and ideally understand some statistics and data mining tools.
Step 2: Find a Business Person. The quickest way to kill an analytics practice is to talk about predictive models, optimization, or statistics with a business person. Instead, find one or more executives who are receptive to testing key assumptions about how the business works. For example, a retail executive might want to know, “Why do some customers stop buying our product?” A social service agency might want to know, “Which spouses are most likely not to pay alimony?” Ask them to dream up as many hypotheses to their questions as possible and then use those as inputs for your analysis.
Step 3: Gain Sponsorship. If step two piqued an executive’s interest, then you have a sponsor. Tell the sponsor what resources you need, if any, to conduct the test. Perhaps you need permission to free up an analyst for a week or two or hire a consultant to conduct the analysis. Ideally, you should be able to make do with people and tools you have inhouse. A good analyst can work miracles with Excel and SQL and there are many open source data mining packages on the market today as well as low cost statistical add-ins for Excel and BI tools.
Step 4: Don’t Get Uppity. “You never want to come across smarter than the executive you are supporting,” says Matthew Schwartz, a former director of business analytics at Corporate Express. Don’t ever portray the model results as “the truth”; executives don’t trust models unless they make intuitive sense or prove their value in dollars and cents. For example, Schwartz was able to get his director of marketing to buy in to the results of a market basket analysis for Web site recommendations because the director recognized the model’s cross-selling logic: “Ah! It knows that people are buying office kits for new employees.”
Step 5: Make It Actionable. A model is worthless if people can’t act on it. This often means embedding the model in an operational application, such as a Web site or customer-facing application, or distributing the results in reports to salespeople or customer service representatives. In either case, you need to strip out the mathematics and decompose the model so it’s understandable and usable by people in the field. For example, a sales report might say, “These five customers are likely to stop purchasing office products from us because they haven’t bought toner in four weeks.”
Step 6: Make It Proactive. The kiss of death for an analytical model is to tell people something they already know. Rather than tell salespeople about customers who are purchasing fewer products than the prior period are likely to churn (like the example I gave in step five above), tell them about customers who will buy fewer products in the future because they have fallen below a critical statistical threshold and are vulnerable to competitive offers. Or, rather than forecast the number loans that will go into default, identify the characteristics of good loans and bake that criteria into the loan origination process. If you deliver results that enable people to work proactively, you’ll become an overnight hero.
Sustaining the Analytics Practice
Let’s assume your initial modeling efforts worked their magic and garnered you strong executive sponsorship. How do you build and sustain an analytics practice? What organizational and technical strategies do you employ to ensure that your analysts are as productive as possible? The following four steps will solidify your analytics practice.
Step 7: Centralize and Standardize the Data. The thing that slows down analysts the most is having to collect data spread across multiple systems and then clean, harmonize, and integrate it. Only then can they start to analyze the data. Obviously, this is what a data warehouse is designed to do, not an analyst. But a data warehouse only helps if it contains all or most of the data analysts need in a format they can readily use so they don’t have to hunt and reconcile data on their own. Typically, analytical modelers need wide, flat tables with hundreds of attributes to create models.
Step 8: Provide Open Access to Data. Data warehouse administrators need to give analysts access to the data warehouse without having to file a request and wait weeks for an answer. Rather than broker access to the data warehouse, administrators should create analytical sandboxes using partitions and workload management that let analysts upload their own data and comingle it with data in the warehouse. This creates an analytical playground for analysts and keeps them from creating renegade data marts under their desks.
Step 9: Centralize Analysts. Contrary to current practice, it’s best to centralize
analysts in an Analytical Center of Excellence under the supervision of a Director of Analytics. This creates a greater sense of community and camaraderie among analysts and gives them more opportunities for advancement within the organization. It also minimizes the chance that they’ll be lured away by recruiters. Although they may be part of a shared services group, analysts should be physically embedded within the departments they support and have dotted line responsibility to those department heads.
Step 10: Offload Reporting. The quickest to undermine the productivity of your top analysts is force them to field requests for ad hoc reports from business users. To eliminate the reporting backlog, the BI team and analysts need to work together to create a self-service BI architecture that empowers business users to generate their own reports and views. When designed properly, these interactive reports and dashboards will meet 60% to 80% of users’ information needs, freeing up business analysts and BI report developers to focus on more value-added activities.
So there you have it, ten steps to analytical nirvana. Easy to write, hard to do! Keep me informed about your analytics journey and the lessons you learn along the way! I’d love to hear your stories. You can reach me at weckerson@tdwi.org.
Posted on 02/26/10 at 7:00 PM2 comments
There is an emerging type of dashboard product that enables power users to craft ad hoc dashboards for themselves and peers by piecing together elements from existing reports and external Web pages. I’m calling these “Mashboards” because they “mash” together existing charts and tables within a dashboard framework. Other potential terms are “Report Portal,” “Metrics Portal,” and “Dashmart.”
I see Mashboards as the dashboard equivalent of the ad hoc report, which has spearheaded the self-service BI movement in recent years. Vendors began delivering ad hoc reporting tools to ease the report backlog that afflicts most BI deployments and dampens sales of BI tools. Ad hoc reports rely on a semantic layer that enables power users to drag and drop predefined business objects onto a WYSIWYG reporting canvas to create a simple report.
Likewise, Mashboards enable powers to select from predefined report “parts” (e.g., charts, tables, selectors) and drag and drop them onto a WYSIWYG dashboard canvas. Before you can create a Mashboard, IT developers need to create reports using the vendor’s standard report authoring environment. The “report parts” are often self-contained pieces of XML code--or gadgets--that are wired to display predefined sets of data or can be easily associated with data from a semantic layer. Power users can apply filters and selectors to the gadgets without coding.
Mashboards are a great way for organizations to augment enterprise or executive dashboards that are designed to deliver 60% to 80% of casual users’ information needs. The Mashboards can be used to address the other 20% to 40% of those needs on an ad hoc basis or deliver highly personalized dashboard for an executive or manager. (I should note that enterprise dashboards should also be personalizable as well.)
Dashmarts? However, there is a danger that Mashboards will end up becoming just another analytical silo. Their flexibility lends themselves to becoming a visual spreadmart, which is why I’m tempted to call them Dashmarts. However, Mashboards that require power users to source all data elements from existing reports and parts should minimize this to some degree.
All in all, Mashboards are a great additional to a BI portfolio. They provide a new type of ad hoc report that is more visual and easily consumed by casual users. And they are a clever way for vendors to extend the value of their existing reporting and analysis tools.
Posted on 02/16/10 at 5:38 AM7 comments
It’s odd that our industry has established a best practice for creating a layer of abstraction between business users and the data warehouse (i.e., a semantic layer or business objects), but we have not done the same thing on the back end.
Today, when a database administrator adds, changes, or deletes fields in a source system, it breaks the feeds to the data warehouse. Usually, source systems owners don’t notify the data warehousing team of the changes, forcing us to scramble to track down the source of the errors, rerun ETL routines, and patch any residual problems before business users awake in the morning and demand to see their error-free reports.
It’s time we get some sleep at night and create a layer of abstraction that insulates our ETL routines from the vicissitudes of source systems changes. This sounds great, but how?
Insulating Netflix
Eric Colson, whose novel approaches to BI appeared two weeks ago in my blog “Revolutionary BI: When Agile is Not Fast Enough,” has found a simple way to abstract source systems at Netflix. Rather than pulling data directly from source systems, Colson’s BI team pulls data from a file that source systems teams publish to. It’s kind of an old-fashioned publish-and-subscribe messaging system that insulates both sides from changes in the other.
“This has worked wonderfully with the [source systems] teams that are using it so far,” says Colson, who believes this layer of abstraction is critical when source systems change at breakneck speed, like they do at Neflix. “The benefit for the source systems team is that they get to go as fast as they want and don’t have to communicate changes to us. One team migrated a system to the cloud and never even told us! The move was totally transparent.”
On the flip side, the publish-and-subscribe system alleviates Colson’s team from having to 1) access to source systems 2) run queries on those systems 3) know the names and logic governing tables and columns in those systems and 4) keep up with changes in the systems. They also get much better quality data from source systems in this way.
Changing Mindsets
However, Colson admits that he might get push back from some source systems teams. “We are asking them to do more work and take responsibility for the quality of data they publish into the file,” says Colson. “But this gives them a lot more flexibility to make changes without having to coordinate with us.” If the source team wants to add a column, it simply appends it to the end of the file.
This approach is a big mindset change from the way most data warehousing teams interface with source systems teams. The mentality is: “We will fix whatever you give us.” Colson’s technique, on the other hand, forces the source systems teams to design their databases and implement changes with downstream analysis in mind. For example, says Colson, “they will inevitably avoid adding proprietary logic and other weird stuff that would be hard to encapsulate in the file.”
Time to Deploy
Call me a BI rube, but I’ve always assumed that BI teams by default create such an insulating layer between their ETL tools and source systems. Perhaps for companies that don’t operate at the speed of Netflix, ETL tools offer enough abstraction. But, it seems to me that Colson’s solution is a simple, low-cost way to improve the adaptability and quality of data warehousing environments that everyone can and should implement.
Let me know what you think!
Posted on 02/08/10 at 12:26 PM1 comments
Special Note: This analysis was written by Philip Russom, Wayne's colleague in TDWI's research department who covers MDM and data integration.
I don’t know about you, but when I read two, similar announcements from competing software vendors, delivered at pretty much the same time, I can’t help but compare them. So that’s what I’m thinking about today (February 3, 2010), after hearing that IBM intends to acquire Initiate Systems. This bears strong resemblance to Informatica’s announcement a few days ago (on January 28, 2010) about their completion of the acquisition of Siperian.
If you work in data management or a related field, these are noteworthy announcements coming from noteworthy vendors, and, therefore, worth understanding. So let me help you by making some useful comparisons – and some differentiations that you may not be aware of. I’ll organize my thoughts around some questions I’ve just seen in the blogosphere and my Twitter deck.
1 – Are both acquisitions about MDM?
Well, sort of. Siperian is well-known for its master data management (MDM) solution. It has attained one of the holy grails among MDM solutions, in that it works with multiple data domains (that’s data about customers, products, financials, and other domains).
Initiate, on the other hand, is well-known for its identity resolution hub. Initiate’s identity resolution capabilities are commonly embedded within applications for MDM and customer data integration (CDI). The way that I think of it, Initiate’s hub isn’t for MDM per se, but it can improve MDM when embedded within it.
At this point, I need to cycle back to Siperian and point out that it, too, provides identity resolution capabilities. And I forgot to mention that Initiate also has some MDM capabilities. You could say that Siperian is mostly MDM, but with identity resolution and other capabilities, whereas Initiate is mostly about identity resolution, but with MDM and other capabilities.
2 - So, the two acquisitions are about identity resolution?
Yes, but to varying degrees. For example, IBMers were very clear that their primary interest in Initiate is its ability to very accurately match data references to patients in healthcare and citizens in government. IBM’s campaign for a Smarter Planet has strong subsets focused on healthcare and government, two industries where Initiate has reference clients doing sophisticated things with identity resolution. My impression is that IBMers are hoping Initiate’s identity resolution functionality will help them sell more products and services into these industries.
Returning to Informatica and Siperian, let’s recall that for years now the Siperian hub has been integrated with Informatica PowerCenter (similar to pre-existing integration among IBM and Initiate products). Among other things, this integration enables Siperian’s identity resolution functions to be embedded within the PowerCenter platform under the name Informatica Identification Resolution. Hence, identity resolution was one off the key capabilities paving the path to this acquisition.
3 – What do these acquisitions mean for IBM and Informatica?
As noted, IBM is counting on the Initiate product to help their campaigns for Smarter Healthcare and Smarter Government. Informatica has now filled the largest hole in its otherwise comprehensive product line, filling it with one of the better tools available via acquisition. Both IBM and Informatica are aggressively building out their portfolios of diverse data management tools, driven both by user demand and competitive pressures. Since both have customers with growing demands for more diverse data management tool types, both will have no trouble cross-selling the new tools to their existing customer bases, as well as selling their older products to the newly acquired customer bases.
4 -- What do these acquisitions mean for technical users?
In my experience, Informatica and IBM both have rather faithful customers, in that they tend to get most of their data management tools from a primary supplier. Technical users from both customer bases now have more functionality available from a preferred technology supplier.
But these aren’t just any data management functions. The two acquisitions focus the spotlight on two of the hottest functions today, in terms of user organizations adopting them, namely: MDM and identity resolution. More than ever, organizations need trusted data, in support of regulatory reporting, compliance, business intelligence, analytics, operational excellence, and other data-driven requirements. MDM and identity resolution are key enablers for these requirements, so it’s no surprise that two leading vendors have chosen to acquire these at this time.
Posted on 02/03/10 at 1:10 PM1 comments
“I love the chart, but what am I supposed to do about it?” With that simple question, Ken Rudin is schooling analysts at Zynga how to deliver information that makes a difference in the way the wildly successful gaming company creates and enhances games for customers.
“My mantra these days is ‘It’s gotta be actionable,’” says Rudin, former CEO of the early BI SaaS vendor LucidEra who now runs analytics at Zynga, creators of Farmville, Mafia Wars, and other popular applications for Facebook, iPhone, and other networks. “Just showing that revenue is down doesn’t help our product managers improve the games. But if we can show the lifecycle with which a subgroup uses the game, we can open their eyes to things they never realized before.”
It’s surprising that Rudin has to do any analytics tutoring at Zynga. Its data warehouse is a critical piece of its gaming infrastructure, providing recommendations to players based on profiles compiled daily in the data warehouse and cached to memory. With over 40 million players and 3TB of new data a day, Zynga’s 200-node, columnar data warehouse from Vertica is no analytical windup toy. If it goes down for a minute, all hell breaks out because product managers have no visibility into game traffic and trends.
Moreover, the company applies A/B testing to every new feature before deploying and has a bevy of statisticians who continually dream up ways that product managers can enhance games to improve retention and collaboration among gaming users. “I’ve never seen a company that is so analytically driven. Sometimes I think we are an analytics company masquerading as a gaming company. Everything is run by the numbers,” says Rudin.
Anticipating Questions
Yet, when Rudin came to Zynga in early 2009, he discovered the analytics team was mostly in reaction mode, taking orders from product managers for custom reports. So, he split the team into two groups: 1) a reporting team that creates reports for product managers and 2) an analytics team that tests hypotheses and creates models using statistical and analytical methods. A third part of his team runs the real-time, streaming data warehousing environment.
The reporting team currently uses a home grown SQL-based tool for creating parameterized reports. Rudin hopes to migrate them to a richer, self-service dashboard environment that delivers most of the routine information that product managers need and the ability to generate ad hoc views without the help of a SQL professional.
Rudin is encouraging the analytics team to be more proactive. Instead of waiting for product managers to submit requests for hypotheses to test, analysts should suggest gaming enhancements that increase a game's "stickiness" and customer satisfaction. “It’s one thing to get answers to questions and it’s another to know what questions to ask in the first place. We need to show them novels ways that they can enhance the games to increase customer retention.”
Zynga is already an analytics powerhouse, but it sees an infinite opportunity to leverage the terabytes of data it collects daily to enhance the gaming experience of its customers. “My goal for the year is to use analytics to come up with new product innovations,” says Rudin. By proactively working with the business to improve core products, the analytics team is fast becoming an ideas factory to improve Zynga’s profitability.
Editor's Note: By the way, the Zynga Analytics team is growing as fast as the company, so if
you’re interested in talking to them, please contact Ken at krudin@zynga.com.
Posted on 02/02/10 at 2:45 PM0 comments
Developers of BI unite! It is time that we liberate the means of BI production from our industrial past.
Too many BI teams are shackled by outdated modes of industrial organization. In our quest for efficiency, we’ve created rigid, fiefdoms of specialization that have hijacked the development process (and frankly, sucked all the enjoyment out of it as well.)
We’ve created an insidious assembly line in which business specialists document user requirements that they throw over the wall to data management specialists who create data models that they throw over the wall to data acquisition specialists who capture and transform data that they throw over the wall to reporting specialists who create reports for end users that they throw over the wall to a support team who helps users understand and troubleshoot reports.The distance from user need to fulfillment is longer than Odysseus' journey home from Troy and just as fraught with peril.
Flattened BI Teams
Contrary to standard beliefs, linear development based on specialization is highly inefficient. “Coordination [between BI groups] was killing us,” says Eric Colson, director of BI at Netflix. Colson inherited an industrialized BI team set up by managers who came from a banking environment. The first thing Colson did when he inherited the job was tear down the walls and cross-train everyone on the BI staff. “ Everyone now can handle the entire stack--from requirements to database to ETL to BI tools.”
Likewise, the data warehousing team at the University of Illinois found its project backlog growing bigger each year until it reorganized itself into nine small, self-governing interdisciplinary groups. By cross-training its staff and giving members the ability to switch groups every year, the data warehousing team doubled the number of projects it handles with the same staff.
The Power of One
Going one step further, Colson believes that even small teams are too slow. “What some people call agile is actually quite slow.” Colson believes that one developer trained in all facets of a BI stack can work faster and more effectively than a team. For example, it’s easier and quicker for one person to decide whether to apply a calculation in the ETL or BI layer than a small team, he says.
Furthermore, Colson doesn’t believe in requirements documents or quality assurance (QA) testing. He disbanded those groups when he took charge. He believes developers should work directly with users, which is something I posited in a recent blog titled the Principle of Proximity. And he thinks QA testing actually lowers quality because it relieves developers from having to understand the context of the data with which they are working.
It’s safe to say that Colson is not afraid to shake up the establishment. He admits, however, that his approach may not work everywhere: Netflix is a dynamic environment where source systems change daily so flexibility and fluidity are keys to BI success. He also reports directly to the CEO and has strong support as long as he delivers results.
Both the University of Illinois and Netflix have discovered that agility comes from a flexible organizational model and versatile individuals who have the skills and inclination to deliver complete solutions. They are BI revolutionaries who have successfully unshackled their BI organizations from the bondage of industrial era organizational models and assembly line development processes.
Posted on 01/27/10 at 9:53 PM5 comments
We’ve come a long way in business intelligence, but there are still plenty of miles to travel. We’ve gone through three distinct eras: Data Warehousing, Business Intelligence, and Performance Management. I think the next era is Decision Analysis.
In the 1990s, we focused on building repositories of integrated, historical data (i.e., the era of Data Warehousing); in the late 1990s and early 2000s, we focused on tools for reporting and analyzing information in our data warehouses (i.e., the era of Business Intelligence); in the late 2000s, we focused on using information to improve performance by monitoring key performance indicators (i.e., the era of Performance Management.)
The next decade will focus on improving the way we make decisions. There is a lot to say here, and I haven’t completely formulated all my thoughts, but this era will take a long time to bear fruit because it involves understanding how the human mind processes information and how people interact in social groups to make decisions. To take BI to the next level, we need better insights into human behavior and perception. In other words, it’s time to recruit psychologists onto our BI teams.
In 2010, you will see the first fruits of the era of Decision Analysis. Specifically, you’ll see more robust collaborative capabilities embedded within BI tools and the first attempts to deliver formalized methods for evaluating the effectiveness of decisions made with those tools.
Collaboration
Most leading BI vendors are applying social media conventions to their toolsets to improve collaboration and decision making. For example, the online-based reporting service, Swivel, lets users rate and comment on charts published online by themselves or others. Following the lead of Facebook, LinkedIn, and other social media sites, some BI vendors will let you “follow” people whose analytical skills you admire and be alerted when they publish a new report. BI vendors will also beef up their guided analytics capabilities, enabling users to review the steps that a trusted analyst took to create a great report using a macro-based replay function. And expect every BI vendor to offer some form of annotation, threaded discussions, and tighter integration with email.
We’ll also see a host of new independent collaboration platforms that could provide the glue to link people, process, and documents in more seamless, transparent ways and improve decision making. For example, SAP is working on an online collaborative environment call 12Sprints that provides templates for specific types of collaboration activities. And Google recently debuted Google Wave, its latest collaborative environment that lets groups engage in seamless instantaneous conversations. Of course, many companies already use Skype, Google Docs, Google Groups, Facebook, and Web conferencing systems, such as GoToMeeting, to foster formal and impromptu collaboration. These incumbents will slowly become more formally integrated with BI tools and decision making processes.
Decision Governance
Although many BI teams do a great job monitoring BI usage, most have done little to nothing to monitor and evaluate the effectiveness of what users do with information they give them. We need to begin tracking the decisions that users make with BI tools and measure the effectiveness of those decisions against business goals and plans. We need to start studying the decision making process and apply procedures to increase the probability that users will correctly interpret the data and take appropriate actions. We can only do this by applying the same types of feedback loops we’ve applied to our BI systems themselves.
A terrorist’s attempt to blow up Northwest Airlines flight 253 last month revealed some fatal flaws in our country’s intelligence gathering activities, including a lack of coordination and information sharing among agencies. But another intransigent problem, it turns out, is the faulty assumptions that analysts apply to evidence and the lack of organizational controls for testing and challenging those assumptions.
A recent article in the Boston Globe called, “Think Different, CIA” provides some instructive lessons for companies using BI tools to make decisions. The article describes a phenomenon that psychologists call “premature cognitive closure” to explain how humans in general, and intelligence analysts in particular, can get trapped by false assumptions, which can lead to massive intelligence failures. It turns out that humans over the course of eons have become great at filtering lots of data quickly to make sense of a situation. Unfortunately, those filters often blind us to additional evidence--or its absence--that would disprove our initial judgment or “theory.” In other words, humans rush to judgment and are blinded by biases. Of course, we all know this, but rarely do organizations implement policies and procedures to safeguard against such behaviors and prevent people from making poor decisions.
The Next Wave
To take BI to the next level, we need to provide a collaborative environment to improve decision making and evaluate the effectiveness of decisions on a continuous basis. We need to establish processes and procedures to ensure people and teams properly interpret the data, identify and challenge each other’s assumptions, and keep an open mind about the drivers of business activity. By applying collaboration and governance to decision making, we can help our companies get even more value from their BI dollars. This really is the next wave of BI.
Posted on 01/19/10 at 9:53 PM3 comments