I went to a small, liberal arts college in western Massachusetts whose mascot is a “purple cow” -- presumably chosen because of the large number of cows that graze in the foothills of nearby mountains that glisten a faint purple as the afternoon sun fades into twilight.
Although our mascot didn’t strike fear in the hearts of our athletic opponents, that was fine by us. We were an academic institution first and foremost. But, it didn’t hurt that our sports teams tended to win more often than not.
We loved our “purple cow” mascot because this bit of serendipity made it hard for others to classify us: most of us so-called "purple people" weren’t just students or athletes or artists but a versatile blend of the three.
I’ve been a “purple person” for some time. And now I want to invite you to be one, too.
Of course, I am not asking you to enroll in my alma mater. I think most of us are too old to qualify at this point! What I am asking, however, is for you to exhibit the versatility of mind and experience that is required to deliver a successful business intelligence (BI) solution.
The Color Purple
The color purple is formed by mixing two primary colors: red and blue. These colors symbolize strong, distinct, and independent perspectives. In the world of BI, let’s say that “red” stands for the BI technologists and “blue” represents the business departments.
In most organizations, these two groups are at loggerheads. Neither side trusts or respects the other. This is largely because neither understands the pressures, deadlines, and challenges that the other faces. And, there is a yawning cultural gulf between the two groups that magnifies their mutual hostility: they speak a different language, report to different executives, travel in different social circles, and possess different career ambitions.
In contrast, a purple person is neither red nor blue. They are neither pure technologist, nor pure business; they are a blend of both. They are “purple people”!
BI requires “purple people” to succeed. Business intelligence is not like most IT disciplines; it requires a thorough and ongoing understanding of business issues, processes, tactics, and strategy to succeed. BI is about delivering information that answers business questions. And since those questions change from day to day and week to week and are often shaped by the larger market landscape, BI solutions can’t succeed unless they continuously adapt.
The only way to create “adaptable systems”-- intrinsically a contradiction in terms--is to find people who are comfortable straddling the worlds of business and technology. These “purple people” can speak the language of business and translate that into terms that IT people can understand. Conversely, they can help business people understand how to exploit the organization’s information repository and analytical tools to solve pressing business problems.
Purple people are key intermediaries who can reconcile business and IT and forge a strong and lasting partnership that delivers real value to the organization.
Finding “Purple People”
So, where can you find “purple people”? Although I’m proud of my college, I wouldn’t suggest that you go there to find them, as smart and ambitious as they might be. In fact, I wouldn’t seek out any college graduates, or by extension, the junior staff of large systems integrators. These folks lack experience in both “red” and “blue” camps. At best, they serve as translators; but lacking real-world knowledge and experience, they always lose something in translation.
The best “purple people” are the proverbial switch hitters. They have strong credentials and a solid reputation in either a business or IT department, and then they switch sides. Their versatility creates an immediate impact.
For example, a financial analyst who joins the BI team brings with him or her knowledge of the finance department and its people, processes, and challenges. They can speak candidly and clearly with former colleagues when hashing out issues and clarifying requirements. They know the difference between “needs” and “wishes” and can create a realistic priority list that works for both sides.
BI directors can straddle both worlds by spending as much time talking with business counterparts as with the technologists on their team. In addition, they should act like business people and manage the BI department like a business: they should craft a strategy document that specifies the department’s mission, vision, values, and strategy and establish metrics for customer success and measure performance continuously.
And, most importantly, BI directors should recruit business people from every department to serve on their BI teams and serve as liaisons to their former departments. These “purple people” form the glue that bonds BI team and business together.
Summary
By definition, business intelligence leverages information technology to drive business insight. As such, pure technologists or pure business people can’t harness BI successfully. BI needs “purple people” to forge tight partnerships between business people and technologists and harness information for business gain.
People often ask me about career opportunities in BI. It should be obvious by now, but my answer is: “Become a purple person. If you straddle business and technology, you are indispensible.”
Posted by Wayne Eckerson on April 29, 20100 comments
A key element in the success of any business intelligence (BI) program lies in the team you create to deliver solutions and the clout it has within the organization to secure resources and get things done. Although there is no right or wrong way to organize a BI team, there are some key principles that are worth knowing. The following the seven guidelines can help you create a high-performance BI team that delivers outstanding value to your organization.
1. Recruit the best people. Jim Collins in his best-selling book “Good to Great” says great companies first get the right people “on the bus” (and the wrong people off it) and then figure out where to drive it. Collins says that the right people will help an organization figure out and execute a strategy, which is a much more effective approach than recruiting people based on their skills and experience to support a strategy which might become obsolete in short order.
From a BI perspective, this means we shouldn’t hire people just because they know a specific tool or programming language or have previous experience managing a specific task, such as quality assurance. If you need specialists like that, it’s better to outsource such positions to a low-cost provider on a short-term contractual basis. What you really want are people who are ambitious, adaptable, and eager to learn new skills. Although you should demand certain level of technical competency and know-how, you ultimately want people who fundamentally believe that BI can have a transformative effect on the business and possess the business acumen and technical capabilities to make that happen.
Collins adds that the right people “don’t need to be tightly managed or fired up; they will be self--motivated by the inner drive to produce the best results and be part of something great.” In a BI setting, these people won’t just do a job; they’ll figure out the issues and work proactively to get things done. Sure, you’ll have to pay them a lot and provide upward career mobility, but a handful of the “right people” will produce more than a dozen “so-so” people.
2. Create multi-disciplinary teams. In most early stage BI teams, a handful of highly motivated people play a myriad roles: the BI project manager serves as the BI architect and requirements analyst; the BI report developer also builds ETL code and perform quality assurance checks; the technical writer provides training and support. With enough talent, resources, and hutzpah, these small, cross-disciplinary teams deliver fantastic results.
And succeeds breeds bigger budgets, more staff, and greater specialization. In most cases, linear development by groups of specialists replaces small, multi-disciplinary teams. The specialist subgroups (e.g., requirements, modeling, ETL, data warehousing, report development, support) operate in isolation, throwing their output “over the wall” to the next group in the BI assembly line. This industrial era approach inevitably becomes mired in its own processes and project backlogs grow bigger. The once nimble, multi-disciplinary BI team loses penchant for speed and loses its luster in the eyes of the business. (See “Revolutionary BI: When Agile Isn’t Fast Enough.”)
The key to remaining nimble and agile as your BI team grows is to recreate the small, multi-disciplinary teams from your early stage BI initiative. Assign three to five people responsibility for delivering an entire BI solution from source to report. Train them in agile development techniques so they work iteratively with the business to deliver solutions quickly. With multiple, multidisciplinary teams, you may need to reset the architecture once in awhile to align what teams are building on the ground, but this tradeoff is worth it.
Multi-disciplinary teams work collaboratively and quickly to find optimal solutions to critical problems, such as whether to code rules in a report, the data model, or the ETL layer. They also provide staff more leadership opportunities and avenues for learning new skills and development techniques. By providing more opportunities for learning and growth, you will increase your staff’s job satisfaction and loyalty. The most productive BI teams that I’ve seen have worked together for 10 to 15 years.
3. Establish BI Governance. Once a BI team has achieved some quick wins, it needs to recruit the business to run the BI program while it assumes a supportive role. The key indicator of the health of a BI program is the degree to which the business assumes responsibility for its long-term success. Such commitment is expressed in a formal BI governance program.
Most BI governance programs consist of two steering committees that meet regularly to manage the BI initiative. An executive steering committee comprised of BI sponsors from multiple departments meets quarterly to review the BI roadmap, prioritize projects, and secure funding. Second, a working committee comprised of business analysts (i.e., subject matter experts who are intensive consumers of data) meets weekly or monthly to define the BI roadmap, hash out DW definitions and subject areas, suggest enhancements, and select products.
The job of the BI team is to support the two BI governance committees in a reciprocal, trusting relationship. The BI team builds and maintains what the committees want to deploy while the BI team educates the sponsors about the potential of BI to transform their processes and what solutions are feasible at what cost. The BI also provides continuous education about emerging BI trends and technologies that have potential to reinvent the business.
4. Find Purple People. The key to making BI governance programs work is recruiting people who can straddle the worlds of business and information technology (IT). These people are neither blue (i.e., business) nor red (i.e., IT) but a combination of both. These so-called purple people can speak both the language of business and data, making them perfect intermediaries between the two groups.
Astute BI directors are always looking for potential purple people to recruit to their teams. Purple people often hail from the business side where they’ve served as a business analyst or a lieutenant to a BI sponsor. Their personal ties to the people in a department, such as finance, coupled with deep knowledge of business processes and data gives them instant credibility with the business. They spend most of their time in the functional area, talking with business leaders and sitting on advisory boards where they share insights about how the business can best leverage BI to accomplish its goals.
5. Aspire to Becoming a Solutions Provider. The best BI teams aren’t content simply to provision data. BI directors know that if the business is to reap the full value of the BI resource, their teams have to get involved in delivering BI solutions. (See “Evolving Your Team from a Data Provider to a Solutions Provider.”) Although some departments may want to build their own reports and applications, few can sustain the expertise to full exploit of DW and BI capabilities to exploit information as a critical resource.
High-performance BI teams work with each department to build a set of standard interactive reports or dashboards that meet 60% to 80% of the needs of casual users in the department. They then train and support each department’s “Super Users” -- tech-savvy business users or business analysts--to use self-service BI tools to create ad hoc reports on behalf of the casual users in the department, meeting the remaining 20% to 40% of their information requirements. The Super Users, in effect, become extensions of the BI team in each department and provide an additional set of “eyes and ears” to keep track of what’s going on from a BI perspective.
6. Give Your BI Team a Name. A name is a powerful thing that communicates meaning and influences perception. Most business people don’t know what business intelligence or analytics is (or may have faulty notions or ideas that don’t conform with the mission of your team.) So spend time considering appropriate names that clearly communicate what your group does and why it’s important to the business.
For example, a group that created predictive models in a large financial services firm called itself the “Marketing Analytics and Business Insights” group. But it discovered that that people didn’t know what the word “analytics” meant or how it differed from “business insights.” Some department heads were resentful of its million dollar budget and the group felt it continually had to defend itself. So it came up with the tagline: “We analyze information to provide usable insights to the organization.” This branding significantly improved the perceived value of the group and it became a “go to” source for information inside the company.
7. Position BI within an Information Management Department. Finally, the BI team should be organized within a larger information management (IM) department that is separate from IT and reports directly to the CIO or COO. The IM department is responsible for all information-driven applications that support the business. These may include: data warehousing, business intelligence, performance management, advanced analytics, spatial analytics, customer management, and master data management.
The IM department maintains a close alliance with the IT department, whose responsibilities include managing the core computing and networking infrastructure that the IM group uses. For instance, the IM group is responsible for designing and managing the data warehouse, while the IT group is responsible for tuning and operating the physical databases that run the data warehouse and the data center in which all data processing occurs.
Separating IM from IT provides a clear signal to the business that these are two separate domains that require different skill sets and career paths. People in the IM group are much more business- and information-driven than those in the IT department, who are more technology focused.
By following these seven steps, you can transform your BI team into a high-performance organization that earns the respect of the business and enjoys sustained success.
Posted by Wayne Eckerson on March 27, 20100 comments
In her presentation on “BI Roadmaps” at TDWI’s BI Executive Summit last month, Jill Dyche explained that BI teams can either serve as “data providers” or “solutions providers.” Data providers focus on delivering data in the form of data warehouses, data marts, cubes, and semantic layers that can be used by BI developers in the business units to create reports and analytic applications. Solutions providers, on the other hand, go one step further, by working hand-in-hand with the divisions to develop BI solutions.
I firmly believe that BI teams must evolve into the role of solutions provider if they want to succeed long term. They must interface directly with the business, serving as a strategic partner that advises the business on how to leverage data and BI capabilities to solve business problems and capitalize on business opportunities. Otherwise, they will become isolated and viewed as an IT cost-center whose mission will always be questioned and whose budget will always be on the chopping block.
Data Provisioning by Default. Historically, many BI teams become data providers by default because business units already have reporting and analysis capabilities, which they’ve developed over the years in the absence of corporate support. These business units are loathe to turn over responsibility for BI development to a nascent corporate BI group that doesn’t know its business and wants to impose corporate standards for architecture, semantics, and data processing. Given this environment, most corporate BI teams take what they can get and focus on data provisioning, leaving the business units to weave gold out of the data hay they deliver.
Mired Down by Specialization
However, over time, this separation of powers fails to deliver value. The business units lose skilled report developers, and they don’t follow systematic procedures for gathering requirements, managing projects, and developing software solutions. They end up deploying multiple tools, embedding logic into reports, and spawning multiple, inconsistent views of information. Most of all, they don’t recognize the data resources available to them, and they lack the knowledge and skills to translate data into robust solutions using new and emerging BI technologies and techniques, such as OLAP cubes, in-memory visualization, agile methods, dashboard, scorecards, and predictive analytics.
On the flip side, the corporate BI team gets mired down with a project backlog that it can’t seem to shake. Adopting an industrialized assembly line mindset, it hires specialists to handle every phase of the information factory to improve efficiency (e.g. requirements, ETL, cube building, semantic modeling, etc.) yet it can’t accelerate development easily. Its processes have become too rigid and sequential. When divisions get restless waiting for the BI team to deliver, CFOs and CIOs begin to question their investments and put its budget on the chopping block.
Evolving into Solutions Providers
Rethink Everything. To overcome these obstacles, a corporate BI team needs to rethink its mission and the way it’s organized. It needs to actively engage with the business and take some direct responsibility for delivering business solutions. In some cases, it may serve as an advisor to a business unit which has some BI expertise while in others it may build the entire solution from scratch where no BI expertise exists. By transforming itself from a back-office data provider to a front-office solutions developer, a corporate BI team will add value to the organization and have more fun in the process.
It will also figure out new ways to organize itself to serve the business efficiently. To provide solutions assistance without adding budget, it will break down intra-organizational walls and cross-train specialists to serve on cross-functional project teams that deliver an entire solution from A to Z. Such cross-fertilization will invigorate many developers who will seize the chance to expand their skill sets (although some will quit when forced out of their comfort zones). Most importantly, they will become more productive and before long eliminate the project backlog.
A High Performance BI Team
For example, Blue Cross/Blue Shield of Tennessee has evolved into a BI solutions provider over the course of many years. BI is now housed in an Information Management (IM) organization that reports to the CIO and is separate from the IT organization. The IM group consists of three subgroups: 1) the Data Management group 2) the Information Delivery group and 3) the IM Architecture group.
- The Data Management group is comprised of 1) a data integration team that handles ETL work and data warehouse administration and 2) a database administration team that designs, tunes, and manages IM databases.
- The Information Delivery group consists of 1) a BI and Performance Management team which purchases, installs, and manages BI and PM tools and solutions and provides training and two customer-facing solutions delivery teams that work with business units to build applications. The first is the IM Health Informatics team that builds clinical analytic applications using reporting, OLAP, and predictive analytics capabilities, and the second is the IM Business Informatics team which builds analytic applications for other internal departments (i.e. finance, sales, marketing).
- The IM Architecture group builds and maintains the IM architecture, which consists of the enterprise data warehouse, data marts, and data governance programs, as well as closed loop processing and the integration of structured and unstructured data.
Collaborative Project Teams. Frank Brooks, director of data management and information delivery at BCBS of Tennessee, says that the IM group dynamically allocates resources from each IM team to support business-driven projects. Individuals from the Informatics teams serve as project managers, interfacing directly with the customers. (While Informatics members report to the IM group, many spend most of their in the departments they serve.) One or more members from each of the other IM teams (data integration, database administration, and BI/PM) is assigned to the project team and they collaboratively work to build a comprehensive solution for the customer.
In short, the BI team of BCBS of Tennessee has organized itself as a BI solutions provider, consolidating all the functions needed to deliver comprehensive solutions in one group, reporting to one individual who can ensure the various teams collaborate efficiently and effectively to meet and exceed customer requirements. BCBS of Tennessee has won many awards for its BI solutions and will be speaking at this summer’s TDWI BI Executive Summit in San Diego (August 16-18.)
The message is clear: if you want to deliver value to your organization and assure yourself a long-term, fulfilling career at your company, then don’t be satisfied with being just a data provider. Make sure you evolve into a solutions provider that is viewed as a strategic partner to the business.
Posted by Wayne Eckerson on March 16, 20100 comments
One of my takeaways from last week’s BI Executive Summit in Las Vegas is that veteran BI directors are worried about the pace of change at the departmental level. More specifically, they are worried about how to support the business’ desire for new tools and data stores without undermining the data warehousing architecture and single version of truth they have worked so hard to deliver.
At the same time, many have recognized that the corporate BI team they manage has become a bottleneck. They know that if they don’t deliver solutions faster and reduce their project backlog, departments will circumvent them and develop renegade BI solutions that undermine the architectural integrity of the data warehousing environment.
The Wisdom of Letting Go
In terms of TDWI’s BI Maturity Model, these DW veterans have achieved adulthood (i.e. centralized development and EDW) and are on the cusp of landing in the Sage stage. However, to achieve true BI wisdom (i.e. Sage stage), they must do something that is both counterintuitive and terrifying: they must let go. They must empower departments and business units to build their own DW and BI solutions.
Entrusting departments to do the right thing is a terrifying prospect for most BI veterans. They fear that the departments will create islands of analytical information and undermine data consistency with which they have worked so hard to achieve. The thought of empowering departments makes them grip the proverbial BI steering wheel tighter. But asserting control at this stage usually backfires. The only option is to adopt a Zenlike attitude and let go.
Trust in Standards
I'm reminded of the advice that Yoda in the movie “Star Wars” provides his Jedi warriors-in-training: “Let go and trust the force.” But, in this case, DW veterans need to trust their standards. That is, the BI standards that they’ve developed in the BI Competency Center, including definitions for business objects (i.e. business entities and metrics), processes for managing BI projects, techniques for developing BI software, and processes and procedures for managing ETL jobs and handling errors, among other things.
Some DW veterans who have gone down this path add the caveat: “trust but verify.” Although educating and training departmental IT personnel about proper BI development is critical, it’s also important to create validation routines where possible to ensure business units conform to standards.
Engage Departmental Analysts
The cagiest veterans also recognize that the key to making distributed BI development work is to recruit key analysts in each department to serve on a BI Working Committee. The Working Committee defines DW and BI standards, technologies, and architectures and essentially drives the BI effort, reporting their recommendations to the BI Steering Committee comprised of business sponsors for approval. Engaging analysts who are most apt to create renegade BI systems ensures the DW serves their needs and helps ensure buy-in and support.
By adopting a Zen like approach to BI, veteran DW managers can eliminate project backlogs, ensure a high level of customer satisfaction, and achieve BI nirvana.
Posted by Wayne Eckerson on February 28, 20100 comments
After 15 years in the business intelligence industry, I’ve hit the mother lode: I’ve discovered the true secret to BI success. It’s really quite simple, and it’s been staring at us for years. It’s the principle of proximity.
By proximity, I mean seating your BI developers next to your business experts. Not just in a joint-application design session, a requirements interview, or scrum stand-up, but ALL THE TIME! Make them work side by side, elbow to elbow, nose to nose. It doesn’t work to merely locate them on the same campus or in the same building. You need to put them in the same cubicle block, or better yet, in one big room with no walls so everyone can see, hear, smell, and touch everyone else all the time. Radical, but effective.
And don’t mistake me: I’m not talking about business requirements analysts--I’m talking about developers who write the code and design the models. Yes, make the developers get the requirements right from the horse’s mouth. Don’t force them to learn requirements second hand through a business requirements analyst. Trust me, something always gets lost in translation.
To develop awesome BI applications, you have to function like a small start up where there are no departments or organizational boundaries, no separatejargon or incentives, no separate managers or objectives, and NO WALLS. Just one big, messy, energetic, on-the-same-wavelength family that gets things done. And fast.
Role of Agile. I like agile software development methods. They come as close as any methodology to approximating the principle of proximity. If nothing else, go agile. Create a small team of business and technical people and make them do stand-up meetings daily, if not hourly! And hold them jointly accountable for the outcome.
But as good as agile can be, proximity is better. Why? When you place developers and business experts in the same room, they almost don’t need to talk. They absorb what they need to know by osmosis, and they learn to respect what each group needs to do to succeed. And fewer meetings make happier, more productive people.
Several years ago, Wes Flores, a technology manager at Verizon, told me the secret of his group’s success: “We sit side by side with business people and report into the same leadership. The only difference is that we specialize in the data and they specialize in the business process.”
So if you want to succeed at BI, reassign your business requirements analysts and immerse your BI developers in the physical heart of the business by applying the principle of proximity.
Posted by Wayne Eckerson on January 7, 20100 comments
It is no secret that many business intelligence jobs are getting outsourced to low-cost centers around the world. In general, these are programming tasks that don’t require direct interaction with customers: ETL programming, testing, and some report development. As offshoring increases, we have to ask, “What are the BI jobs of the future and who will fill them?”
Drive the Business. A column by Thomas L. Friedman wrote a column for the New York Times this fall (October 21, 2009) that sheds some light on the issue:
“A Washington lawyer friend recently told me about layoffs at his firm. I asked him who was getting axed. He said it was interesting: lawyers who were used to just showing up and having work handed to them were the first to go because with the bursting of the credit bubble, that flow of work isn’t there. But those who have the ability to imagine new services, new opportunities and new ways to recruit work were being retained. They are new untouchables.”
“That is the key to understanding our full education challenge today. Those who are waiting for this recession to end so someone can again hand them work could have a long wait. Those with the imagination to make themselves untouchables—to invent smarter ways to do old jobs, energy-saving ways to provide new services, new ways to attract old customers or new ways to combine existing technologies—will thrive.”
BI Imagineers. The good news for BI professionals is that it truly takes imagination to deliver an effective BI application. BI needs people who can stand between the business and technology and create solutions that anticipate user requirements for information, who help create and monitor metrics that drive performance, and help the business leverage information technology for competitive advantage.
Those individuals who can help their organizations harness technology for business gain will be in high demand and garner substantial salaries. But their knowledge can’t be gained easily—it takes years of working in the field applying technology to business problems before practical experience translates into imaginative solutions that drive the business. Imagination requires not just technical literacy—that’s just the ticket to play the game—but it takes deep knowledge of the business, its goals, strategy, people, and processes. Acquiring that knowledge takes time.
Apprenticeships Needed. I fear that if we offshore all the low-skill, entry-level BI jobs, people will never get the apprenticeships they need to become BI imagineers (to borrow a phrase from Disney.) How will people gain a foothold in the industry if there are no entry level jobs?
Jim Gallo, a consultant at Information Control Corp (ICC) in Columbus, Ohio wrote a provocative article on this topic for the BI Journal which I highly recommend. He writes, “It’s simple, really: Unless CIOs, CFOs, and CEOs make a commitment to provide opportunities to BI neophytes, we all run the risk that our BI organizations will cease to exist as strategic enablers within our own organizations.” (TDWI members can log in and read the article here.)
Gallo’s company has figured out an efficient and effective way to hire and train college graduates and make them productive members of an agile BI project team. In the article, Gallo discusses how these blended teams—which are comprised of three junior developers, a senior architect and a senior QA analyst—can compete cost-effectively against offshore BI players.
With such an apprenticeship, the junior developers on ICC’s agile teams are well on their way to becoming the BI leaders of tomorrow, garnering well-paying jobs. They are honing their technical skills by solving real customer problems under the guidance of senior BI architects and analysts. We need to make such opportunities available in our BI programs to create the imaginative leaders of tomorrow (if not today!)
Posted by Wayne Eckerson on December 28, 20090 comments
It’s funny how we rarely eat our own dogfood. Someone recently asked a question in TDWI’s LinkedIn group about the key performance indicators that data warehousing managers should use to measure BI/DW effectiveness. The first several respondents cited classic IT metrics, such as data transfer volume, data sizes, average ETL run times, average query response times, number of reports, number of users, etc. Then, a few people reminded everyone that KPIs, by definition, are “key”—typically, few in number, driven by top-level goals, easy to understand, and actionable.
The best KPIs drive dramatic improvements to the business. As such, there may only be one or two true KPIs per function (e.g. BI/DW) per level: operational, tactical, and strategic. Typically, each set of KPIs drives performance in the level above. Thus, operational KPIs in an operational dashboard drive tactical KPIs in a tactical dashboard which drive KPIs in a strategic dashboard. Each dashboard is used by different people at different levels of the BI/DW organization--from analysts and administrators to managers and architects to directors and sponsors--but the aligned KPIs ensure that everyone is working towards the same end.
For example, an operational dashboard measures processes in flight that enable ETL analysts, BI analysts, or database administrators to fix problems as they happen and optimize performance. At this level, KPIs for BI/DW might be: “The ETL job is spitting out a large number of errors.” “Query response times are slow.” Or “The number of users on the system is below normal.” In each case, an ETL developer, database administrator, or BI analyst respectively takes immediate action to investigate and fix the source of the problem.
A tactical dashboard measures departmental objectives and goals. In the BI/DW world, these goals are specified in service level agreements established with data warehousing customers. These might include DW availability (“The DW is refreshed at 8 a.m. week days 90% of the time”) and DW reliability (The “DW is operational 99.99% of the time between 8 a.m. and 5 p.m. Eastern time.”) These KPIs show whether the BI/DW team is on track to meet SLAs in a given time period. If not, the BI manager or architect needs to figure out why and take action to meet SLAs before it's too late.
A strategic dashboard measures how well the BI/DW team is achieving its strategic objectives. Say for example, the BI/DW team’s mission is to “Empower business users with timely, reliable data that improves decisions.” A strategic KPI could be "the number of unique BI users each week who run a report," or the "ratio of active vs inactive users each week," or the "percentage of users rating BI as “critical to making quality decisions this period'” in a semi-annual survey. Lower than expected usage should cause BI directors and sponsors to understand what inhibits usability and take steps to improve it.
Sometimes, a single strategic KPI from the BI/DW team lands on the scorecard of senior executive. The executive uses the KPI to judge the effectiveness of the BI/DW resource and make funding decisions. Thus, it’s really important to think long and hard about that single KPI and how all your efforts are focused toward achieving positive results there. This can be done if your BI/DW KPIs are aligned so that lower-level KPIs drive higher-level ones and all are based on measurements that drive actions.
Let me know what you think!!
Posted by Wayne Eckerson on December 10, 20090 comments
In a prior blog, I discussed strategies for crossing the “Chasm” in TDWI’s five-stage BI Maturity Model. The Chasm represents challenges that afflict later-stage BI programs. In my prior blog, I showed how later-stage BI programs must justify existence based on strategic value rather than cost savings efficiency, which is a hallmark of early-stage BI programs.
But perhaps a more serious challenge facing BI programs that want to cross the Chasm is migrating from a departmental BI solution to one with an enterprise scope.
Departmental Solutions. In my experience, some of the most successful BI solutions in terms of user satisfaction are developed at the department level. One reason is because the scope of the project is manageable. A department is small enough so that the technical developers might sit side by side with the business analysts and subject matter experts. They may see themselves as belonging to a single team and may even report to the same boss. The BI project is smaller and probably draws from just one or two sources that are used regularly and well known by people in the department. Moreover, hammering out semantics is straightforward: everyone uses the same language and rules to define things because they all live and work in the same “departmental bubble.”
Chasm Challenges
Politics. Conversely, developing an enterprise BI solution or migrating a successful departmental one to the enterprise level is fraught with peril. The politics alone can torpedo even the most promising applications. The departmental “owners” are afraid corporate IT will suck the lifeblood out of their project by over-architecting the solution. Corporate sponsors jockey for position to benefit from the new application but are reluctant to pony up hard dollars to lay the infrastructure for an enterprise application that benefits the entire company, not just them.
Semantics. In addition, semantics are a nightmare to resolve in an enterprise environment. What sales calls a customer is often different than how finance or marketing defines a customer. Ditto for sale, margin, and any other common term used across the enterprise. In fact, it’s always the most common terms that are the hardest to pin down when crossing departmental boundaries.
Scope. Finally, scope is an issue. Enterprise applications often try to tackle too much and get stalled in the process. The number of data sources rises, the size of the team increases, and the number of meetings, architectural reviews, and sign offs mount inexorably. The team and its processes eventually get in the way of progress. Simply, the BI program has become a bottleneck.
Hybrid Organizational Model
To cross the chasm, BI programs need to maintain the agility of departmental scale efforts with the economies of scale and integration delivered in enterprise solutions. To do this teams must adopt a hybrid organizational model, in which the central or corporate BI team manages the repository of shared data, a common semantic model, and a set of documented standards and best practices for delivering BI solutions.
Departmental IT. The corporate team then works closely with departmental IT staff responsible for developing BI solutions. The departmental developers leverage the enterprise data warehouse, common semantic model, and ETL and BI tools to create subject-specific data marts and standard reports tailored to departmental users. The corporate team oversees their development, ensuring adherence to standards where possible.
Super Users. If no IT staff exists in the department, the BI team cultivates “super users”—technically savvy business users in each department—to be their liaison or “feet on the ground” there. The corporate BI team creates subject-specific data marts and standard reports for each department using input from the Super Users. It then empowers the Super Users to create ad hoc reports around the boundaries of the standard reports but not overlapping them.
BICC. A fully-functioning BI Competency Center adopts this hybrid model where the corporate BI staff work cooperatively with departmental IT professionals and super users The glue that holds this hybrid model together are the standards and best practices that empower departmental experts to build solutions quickly that leverage the enterprise resources.
This approach ensures both agility and alignment, a rare combination seen in most BI programs. The fear of giving control back to departments and proliferating spreadmarts is why many companies are stuck in the chasm. But unless the corporate BI team cedes development work via well documented standards and practices, it becomes a bottleneck to development and causes spreadmarts to grow anyway.
Posted by Wayne Eckerson on November 16, 20090 comments
TDWI’s Maturity Model is represented by a bell curve spanning five stages of maturity. The curve, which represents the percentage of companies at each stage, is broken in two spots. The first is the Gulf and the second is the Chasm. (See figure 1.)
The Gulf and Chasm represent the series of obstacles and challenges that early- and later-stage programs encounter during their BI journey, respectively. Companies in the Gulf struggle with sponsorship, funding, data quality, project scope, and spreadmarts. Companies in the Chasm struggle with the politics, logistics, and dynamics of delivering an enterprise BI environment. (An enterprise environment may span the entire organization, a business unit, or profit-loss center.)
While the Gulf is difficult, the Chasm is downright perilous. Many BI programs fall into the Chasm and are never heard from again. During the next several weeks, I’ll write a series of blog posts designed to help BI professionals better understand the hazards of the Chasm and how to overcome them.
Figure 1.
From Cost-Savings to Strategic Value
The first major challenge that companies face in the Chasm is re-justifying their existence. By now, the BI team has squeezed all the cost-efficiencies out of the reporting process and must justify itself by the strategic value it offers the organization. Executives funded early data warehousing projects in part because of the projected ROI from consolidating legacy reporting systems and freeing high-priced analysts from having to manually create standard weekly or monthly reports.
Now, the BI program must do more than reduce costs and streamline IT processes. It needs to become a mission-critical resource that executives see as critical to the company’s growth. This is especially true if BI represents a sizable portion of the IT budget and employs more than a handful of full-time staffers. With a six- or seven-figure budget, the BI program has a huge bulls-eye on its back during budget season. To address the question, “What has BI done for us lately?” the BI team must have a ready and compelling answer.
Astute BI directors craft a vision for BI in business language and peddle it constantly. More than pitching proverbial “single version of truth,” they discuss how BI is critical to understanding the profitability of every customer or every product and how this information drives critical decisions about how the company invests its time, resources, and money. They talk about how BI is critical to achieving a 360-degree view of customers or suppliers and how that information has increased revenues and decreased procurements costs, respectively.
Savvy BI directors also discuss how BI is bridging the barriers between departments. For example, they talk about how finance directors and operations managers can speak the same language because general ledger data is aligned with product sales at a detailed level. They also show the company’s performance has improved since the BI team started displaying metrics with detailed, supporting data within easy-to-use dashboards and scorecards.
Talking the Talk
Of course, BI managers won’t have much to talk about if they haven’t done these or other strategic initiatives. But at least they can relay the vision they are working on, and bank on goodwill until they have tangible results to show.
But even “talking the talk” can be a stretch for some BI managers who grew up in IT with a technological perspective of the business. Thus, the first key to crossing the chasm is understanding the business and developing a strong rapport with business executives and managers by framing BI in their context.
Stay tuned for a description of the other major challenges facing companies in the Chasm in future posts.
Posted by Wayne Eckerson on October 19, 20090 comments
There is a dirty, little secret about data warehouses: we wouldn’t need them if top executives ran their organizations properly.
A data warehouse (DW) reflects the organization—the more fractured and disintegrated the organization, the harder it is to create a robust, highly functional data warehouse. These reporting repositories really are tools to reintegrate a fractured enterprise and provide a holistic and consistent view of data where none exists.
Most organizations are like Humpty Dumpty teetering and tottering on top of a big wall. With the slightest gust of wind, Humpty crashes and breaks into dozens of pieces. And DW teams are “all the king’s horses and all the king’s men” who are charged with putting Humpty Dumpty back together again.
Today, most companies have fragmented into dozens or hundreds of largely unconnected business units, departments, and workgroups, each with their own strategies, policies, processes, systems, IT staffs, and data. It’s the job of the CEO to bring order to this chaos. If the CEO provides the business with a clear, coherent strategy, integrated processes and systems, and standard definitions and rules governing those processes, then there is almost no need for a data warehouse.
The good news is that many DW teams do succeed in gluing their companies back together, at least for awhile, until the next merger, acquisition, or other upheaval blasts everything apart. But while it lasts, these unsung heroes should be congratulated, not outsourced, offshored, or reassigned to the IT-equivalent of a Siberian gulag.
The moral of the story is this: don’t blame the DW team if it can’t deliver a successful data warehouse; blame the CEO. The data warehouse is merely a messenger, a reflection of the state of organizational dysfunction.
Editor's note: this article originally appeared in the Washington Post.
Posted by Wayne Eckerson on September 23, 20090 comments