A Phased Approach to Data Warehouse Projects
Massive waterfall solutions may not be the right approach for many enterprises. We offer a better way to build data warehouse success.
By Richard Brayshaw
One of the benefits of being an independent consultant in data warehousing, management information (MI), and business intelligence is the opportunity to see similar (or even identical) problems tackled in often quite different ways. There is seldom one best solution to a particular MI/BI problem.
Many factors come into play. The best technical solution may not be the most cost-effective solution, the fastest solution may not be the best long-term solution, and so on. Although the problems to be solved in a particular industry sector are often similar, the solution has to fit with the individual organization’s operational and financial parameters and not aim to satisfy some lofty, academic ideal of architectural and functional perfection.
Pragmatism and flexibility have to be the watchwords. What usually happens is that the “best” solution from a technical and academic point of view is gradually “degraded” in order to fit within time, cost, and other limiting parameters.
Take the example of an organization that, for one reason or another, has fallen behind the market median in its ability to collect and disseminate management information in a timely and accurate manner and now wishes to catch up and regain some of the competitive advantages ceded to its competitors in recent years. That company may have other drivers for change: regulatory compliance, reduced administrative headcount, retention of key sales staff, and so on. It may have a limited budget to achieve its ambitions. It may also wish to transition to its Brave New World of MI superiority within quite tight timescales.
The thought of some IT consultants pitching a multi-million dollar (or pound or Euro) IT system may not go down terribly well at a board of directors meeting. Such a pitch may dissuade the executives holding the purse strings from proceeding at all!
Obviously, for this organization, a massive, multi-year, full-specification MI project is not going to be the correct fit. The organization must be persuaded and advised as to the most cost-effective course of action. An initial scoping phase will determine exactly what data and what information is the most important and most urgent and must focus on the design of a system for collecting and disseminating just this.
The first stage is, then, a compromise, but one which, when delivered in a short enough timescale, shows the promise of a phased approach and builds confidence for the future. Crucially, it serves to educate a non-technical group of stakeholders in the fundamentals of MI. We get to show off our colorful toolsets, high-speed information retrieval, and never-before-seen combinations of data and also give a hint of the potential to be released by pushing on further.
Your client loves to know more about their own organization; they crave the information they need to manage and direct and decide. If you can persuade them to let you analyze and categorize their data and the associated administration processes and to suggest to them how savings can be made and how decisions can be supported by the collection, combination, and reporting of the data that <em>they already own</em>, you will have their attention. If you can persuade them that this process need not take years before any results are seen and will begin to bear fruit within months, then all the better.
Offering a road map for future expansion helps enormously. A steady delivery of MI functionality to the business demonstrates progress, spreads the costs, and builds trust. Once the stakeholders see that the MI project is delivering real benefit (albeit in a limited area), they will be in a better position to commission further expansion across the organization. In fact, once word gets out about the early successes, other departments and business functions will lobby to have their areas supported by the new system. I’ve seen it happen many times and it is a joy to witness!
The road map should build out from the first phase of the data warehouse in logical tranches, each of which adding “more-than-the-sum-of-its-parts.” For example, if the first phase is the addition of gross sales figures to the data warehouse and the second phase is the addition of costs, then the calculation of net sales profit is an obvious additional development and a demonstration of the potential inherent in the combination of previously unrelated data in the data warehouse. Previously, the task of bringing such figures together may have been a cumbersome and costly manual process. Now it is performed in a controlled and fully automated fashion on a regular basis. Administration overheads are reduced and return on investment is realized.
Business owners and operators are rarely versed in the dark arts of MI and data warehousing. Even their in-house IT departments may have little or no experience in these areas. Going in feet first with a proposal for a massive waterfall solution might just scare the living daylights out of everyone. Instead, demonstrate, prototype, and deliver in phases. Build trust and show progress with smaller, quicker, cheaper steps that form part of a wider vision. They’ll thank you for it.
- - -
Richard Brayshaw is the MI services lead at Atticus Associates Ltd in the City of London. Richard has over ten years’ experience in data warehousing analysis and design in the UK banking and insurance sectors. You can contact the author at [email protected].