TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Avoiding Pitfalls in Data Architecture Shortcuts

Do "Band-Aid" approaches always seem to become the permanent solution? You need to avoid shortcuts to ensure a strong foundation in your data architecture.

• By Wes Flores
• August 10, 2016

As project leaders, we often face difficult decisions determining the best approach to take for the data architecture of a project. When time, budget, or resource constraints take priority over quality, project teams may assume that shortcuts should be taken for the greater good.

Data architecture is foundational to your business intelligence and analytics programs. It defines how data is structured and integrated, so its integrity should not be compromised.

One justification that you often hear about a shortcut is that the solution will only be temporary, but this is hardly ever the case. I often say: "The Band-Aid approach always becomes the permanent solution."

Four Reasons Data Architecture Shouldn't Use Shortcuts

-- It is critical to get data architecture and other foundational aspects of data management right because future projects will depend on them.

One example of this is your data quality process. If data quality is not a foundational aspect of the architecture, the system will maintain the original state of the data, including any poor quality data. All future projects will inherit the poor quality data and have to individually address the resulting challenges.

-- People will inevitably adapt to the limitations the shortcuts impose by implementing inadequate and costly work-arounds.

This can apply to many situations, but one example would be an enterprise data warehouse where the needed data is not integrated in a meaningful way. Users will start to develop duplicate table structures and may even build their own local version of the data to achieve their integration needs.

In the end, you not only have workarounds, you have many different workarounds, each built by an individual user without leveraging anyone else's efforts.

-- It is extremely hard to change a data architecture solution after the fact due to its foundational nature.

Take a practical example: field names. Due to timeline constraints, the working team may not take the time to build a good naming convention or adhere to the one in place. Updating and standardizing the field names might not be discussed until developers and users have problems working with the data. After the fact, however, changing field names across known and unknown processes comes with many risks, making even simple changes very costly.

-- Budgets are rarely increased after the fact to address cleanup.

One example I recall was a product launch that required strategic metrics for the new product in two weeks. Due to the time constraint, the project team decided not to build this out in the standard way that other metrics were created. A quick decision was made to build a temporary table to enable metrics to be available day one of the launch. The business need was addressed; however all future changes continued on the shortcut structure.

The next year, IT requested funding to integrate the reporting into the standard metrics process to decrease future costs of maintenance and improve consistency. The business understood the value of the request, but because the critical needs were being met, they were not willing to fund the improvement.

The Reality of Many Data Architectures Today

If shortcut approaches continue to be taken, the proper data architecture will never be created. After many years of compromises, your data architecture will be so convoluted and complex that it will need a complete overhaul.

Simply put, building on top of a poor foundation will create poor outcomes and result in the propagation of issues in your information delivery systems. These small compromises add up over time, creating additional compromises on dependent and downstream systems, ultimately impacting your ability to improve the customer experience.

How to Reduce Shortcuts in Your Data Architecture

When faced with the shortcut dilemma in your next project, remember that bad data architecture is built one bad decision at a time -- therefore every decision counts!

Of course, that can be easier said than done. Here are five practical ideas to help reduce the likelihood of shortcuts being taken or maintained.

1. Before taking issue with an approach, perform a cost-benefit analysis and first prove the pros and cons to yourself. This will help you see both sides of the issue and strengthen your case to show that a shortcut solution will indeed cost more in the long run.

2. If your company has an enterprise architecture team, bring them into the mix early as they typically have a broader perspective and will lend some additional weight to your argument.

3. Perform an analysis of other shortcut approaches implemented over the past few years. If you find a pattern of issues, this can be an effective communication tool to help people see the impact of these decisions over time.

4. If the immediate business need requires a shortcut solution, consider breaking the project into two phases with the initial phase providing the required shortcut and the second phase providing the better long-term solution. By confirming a second phase and budget up front, you will ensure you are not left with a Band-Aid solution.

5. If you have inherited a project or solution flawed by shortcuts, you can still make a difference. Be proactive by pushing for revision projects with your project management office or getting business owners to agree to fund revisions in the next year. Follow the appropriate process -- depending on how projects are managed, prioritized, and funded in your business -- and drive for the needed improvements.

Using these tactics, you may find that fewer shortcuts will be proposed in your future data management projects. Ideally, modeling this type of leadership will demonstrate your understanding of a balanced and broad perspective.

When it comes to data architecture, you can confidently defend doing it right the first time, for only then will it stand the test of time.