Between analytical modelers and IT administrators is a cultural divide.
Welcome to TDWI FlashPoint. In this issue, Wayne Eckerson offers five steps to help you bridge the divide between analytic modelers and IT administrators.
- FlashPoint Snapshot
BI Search and Text Analytics: New Additions to the BI Technology Stack
Five Steps to Bridge the Business/IT Gap
- FlashPoint Rx
Don't Take a Myopic View of ETL Usage
FlashPoint Snapshots highlight key findings from TDWI's wide variety of research.
Vendor Products Related to BI Search and Text Analytics
The list is representative and not meant to be comprehensive. (View larger image.)
Source: BI Search and Text Analytics: New Additions to the BI Technology Stack (TDWI Best Practices Report, Q2, 2007). Click here to access the report.
Based on 370 Internet-based respondents. Rounding and multiple-choice questions are responsible for percent totals that do not equal exactly 100 percent.
Five Steps to Bridge the Business/IT Gap
Wayne W. Eckerson, TDWI
There is a cultural divide between analytical modelers and IT administrators. Anyone who has managed or mediated problems between these two groups knows how different they are -- culturally, organizationally, and functionally.
Analytical modelers view IT as gatekeepers to the data and bottlenecks to getting things done. They see IT as paralyzed by process and blinded by methodology. From the perspective of the IT department, on the other hand, analytical modelers are CPU hogs who issue runaway queries and kick off CPU-intensive scoring routines that kill query performance on the database server and slow query response times for everyone.
Most of the solutions for bridging the divide are organizational and cultural in nature. In other words, the solution lies in changing attitudes and behaviors, and this type of change doesn't come easily or quickly.
There are five ways to bridge the divide between IT and analytical modelers.
- Find a Liaison. Every organization that has successfully aligned the two groups has had one or more liaisons who straddle both worlds -- they can talk the language of business and IT and align the two groups through force of personality and partnerships.
- Foster Dialogue. Bring the two groups together so that they can begin to understand each other and overcome prejudices and knee-jerk reactions. This can be done by 1) having the groups socialize outside of work, 2) conducting formal meetings where the two sides air their grievances and acknowledge common objectives, and 3) establishing formal channels for regular communication between executives and managers in the two groups.
- Compromise. Once the two sides understand each other better, it's time for both to roll up their sleeves and change their habits to accommodate the fundamental needs of the other group. Often, this means that analysts should learn the language of IT, such as SQL, and adopt some IT processes, such as modeling rules and naming conventions, especially if they want IT to deploy their models in a quick and efficient way. Conversely, IT needs to give analysts access to the corporate jewels (i.e., the data warehouse) and give them space within the database server to add, delete, and manipulate tables and data as they see fit without being restricted by storage capacity, CPU threads, or memory.
- Training. To make the changes just described requires that both groups pick up new skills. Analysts need to learn how to write SQL and create and manipulate tables in the database. Database administrators need to learn how to configure and tune a mixed workload environment that supports a combination of casual users, power users, and analysts. This may require hiring consultants to show the database administrators how to configure a mixed workload system in their own environment.
- Establish an Analytic Sandbox. These dedicated environments let analysts do all the things analysts love to do. This way, they can create tables; load, merge, and aggregate data; combine fields; and run queries -- without impinging on other users of the data warehousing environment.
- An analytic sandbox partitions spaces within a data warehousing database, giving modelers read-only access to any data in the warehouse so they don't have to download it to a data mart or desktop workbench. Using partitions and workload management utilities, IT can also give modelers free reign to load and manipulate data in the sandbox without affecting performance for other users of the system or jeopardizing the integrity of data within the data warehouse. Recent advances in database technology -- specifically workload management -- have made this an increasingly popular option.
- Virtual sandboxes require a database that supports partitions and robust workload management. This can be challenging for database administrators who don't know how to configure a mixed workload environment or whose database environment isn't powerful enough to support dozens or hundreds of partitions for analytical modelers. As a result, companies may need to spend money to hire consultants and upgrade their database environment to support analytic sandboxes.
To overcome the cultural and organizational divide and get these groups working together harmoniously for the good of the organization, business and technology managers should find a liaison, foster dialogue, compromise, train, and implement analytical sandboxes.
FlashPoint Rx prescribes a "Mistake to Avoid" for business intelligence and data warehousing professionals from TDWI's Ten Mistakes to Avoid series.
Ten Mistakes to Avoid When Selecting and Deploying ETL Tools
Mistake 3. Taking a Myopic View of ETL Usage
What does the future look like for your environment? From a data perspective, this may mean dealing with time-sensitive data. At first, you might process data in a nightly batch, but chances are good that you will need to add low-latency data in the near future, or provide on-demand access to just-in-time BI tools or dashboards. If you don’t think about such future needs, then you may ignore features like messaging, EAI or Web service integration, or the broader set of data federation features provided by EII tools or components.
When evaluating ETL tools, it also pays to look beyond basic ETL to broader data integration needs. For example, use beyond a single project will drive the need for multi-project source control and security -- features you wouldn’t consider if your scope is a single data warehouse.
Many organizations are realizing how useful ETL tools can be outside the data warehouse environment. The tools are good at getting data from one place to another, which means ETL tools can be a great fit for system migrations and consolidations.
Moving data from one application to another or merging several different applications into one system is a lot of work, particularly if it involves dealing with packaged applications like SAP. These data migration efforts are often underestimated. If you are having trouble justifying an ETL purchase based on data warehouse savings, take a step back and look at reuse outside of the data warehouse. You may find that other projects have similar needs, which could provide the ROI you need to justify your purchase.
This excerpt was pulled from the Q2 2006, TDWI Ten Mistakes to Avoid
series, Ten Mistakes to Avoid When Selecting and Deploying ETL Tools,
by Mark Madsen.