Why Using Principles of Federalism Is the Future of Data Warehousing
Data warehousing isn’t dead. Automated data warehousing applies the principles of America’s Founding Fathers to enhance data democratization.
- By Dan Bruns
- August 28, 2020
When federalism was implemented in the United States in late 1787, power was divided between the federal government and the states of the Union. The Founding Fathers realized that each state had different needs. Except for instances where the federal government was given specific responsibilities, the states were able to govern themselves.
When applying this same idea to an enterprise data warehouse, organizations have an opportunity to enhance the security and efficiency of their processes. Imagine each constituent department as a state, completely in control of its own data from procurement through aggregation, including the integration of its department-specific business rules. These constituencies band together to create the union -- an enterprise data warehouse.
Each department could have its own data-savvy business personnel working directly with the department’s data, building aggregations the department cares about, for example:
- Marketing: 360-degree views of customers
- Finance: Profit and loss
- Sales: Sales data by product and region
- HR: Employee business and travel expenses
Data could be shared seamlessly among groups, just as states work with each other on projects. For example, HR could easily and securely share its authoritative list of employees within the company and have control over who can access it. Plus, similar to the U.S. government structure, each state (that is, department) could send representatives to a forum to debate data usage, form cooperative efforts to bring in new data sources, and enforce data governance.
With this business architecture, the IT group’s role would change, and that would be a welcome development for people within IT. Traditionally, IT bears the brunt of the work required to build data warehouses because IT is where a business consolidates its expensive and specialized ETL engineers.
Under that structure, technical IT people need to become experts in translating vague business requirements into real data, ensuring business processes (with which IT has no experience) mirror the data output. Errors are often encountered with this translation, and IT is on the hook to repair data flows, often in response to a furious business partner.
In the new world, IT would no longer need to explain why calculations were developed a certain way -- they would just refer the question to the business group directly. This gets IT out of the data management role they have been thrust into and back to doing what IT does best:
- Taking care of the infrastructure, ensuring data queries are optimized to return data expeditiousl
- Curating core data sources (where applicable) from which they allow their departmental colleagues access within their respective areas
- Laying down overarching “laws” for the governance of data and overseeing its integration and overall use
IT managers might be wondering in this scenario why a business would populate each area with its own ETL staff members (which can be perceived as a waste of money because you’re duplicating roles instead of housing all the resources within IT). Although there were good reasons why businesses historically consolidated their ETL staff members and contractors into the IT department, times have changed. Now we have a new breed of tools for data warehousing assistance: data warehouse automation tools.
Today, data warehouse automation tools are mature enough to augment or replace data warehousing resources within each business area, enabling existing business-side staff members who know their data, but might not have data warehousing experience, to build and maintain their own data warehouse or data mart. The software uses its own intelligence to translate the business’s metadata into facts, dimensions, and other data warehousing structures. The results are created directly from the business’s efforts, as opposed to a game of telephone with IT when a business unit wants new data sources integrated.
This arrangement will get data into the hands of the people who use it most much faster than previous methods. Similar to the Constitution, this federalized data arrangement is the first step in creating a more “perfect union” -- bringing together every part of the business on the path to a stronger, more data-rich future. By using technology and automation to put people in the best position to govern their departmental data and contribute to the overarching business in a more harmonious way, everyone wins.
About the Author
Dan Bruns is the founder and president of Pyramart where he is responsible for driving the strategy of automated data warehousing. You can reach the author via email or on his website at Pyramart.com.