Your Tech Debt and Business Analytics Blind Spots are Bigger Than You Think
IT leaders need to actively shine a light on all parts of your tech/business gap -- from project backlogs to business expectations -- to secure additional resources.
- By Stan Pugsley
- January 29, 2020
The technology gap at every enterprise looks different depending on who is describing it. To a business executive focused on business analytics, the technology gap includes all the capabilities and services they would like (for their own use or to offer to employees or customers) but which they haven't yet been able to buy or develop internally. For example, they may wish to see advanced reporting or analysis tools, automated data integration of new data sources (to allow more frequent data refreshes), or a new portal where customers can see up-to-the-minute information about their account or services. Business executives may be evaluating a new product, merger, or acquisition and feel hamstrung by the lack of reliable data to evaluate the opportunity or to make the change occur. Most frequently of all, business groups lack trust in the quality or authority of their available data.
To a technology team, the technology gap consists of projects in the backlog or queue they haven't had the time or resources to complete. The tech team would also have a list of tech work that they may not be publicly advocating but know they need to tackle at some point, such as software upgrades, documentation, security fixes, or re-architecting projects (such as for the cloud).
This gap may be the source of insecurity because technology leaders feel perpetually understaffed and overwhelmed by incoming requests. However, viewed another way, clearly understanding and documenting the gap provides a powerful way to align priorities with sponsors and make a case for additional budget and resources.
When business and technology groups have a different view of this gap, it becomes difficult to prioritize projects and budget for future needs.
In the 1950s, psychologists Joseph Luft and Harrington Ingham developed a communication model called the Johari Window to help different parties better understand each other and their shared needs. The Johari Window consists of four sections:
Open Area contains elements that are known to ourselves and to others. This is shared, understood ground.
Blind Spot consists of things that are important to others but are hidden or neglected by us.
Hidden Area is also called the façade. It is filled with things we don't share with others, either intentionally or unintentionally.
Unknown is filled with ideas that neither party has explored but that could be the source of hidden or future friction.
Figure 1: The four quadrants of the classic Johari Window.
This model can be adapted to the relationship between technology teams and their business stakeholders to help close the technology gap and improve communication between parties. In this new version of the model, shown in Figure 2, the sections consist of:
Open Backlog with a prioritized list of tasks and projects that all parties agree to pursue. We want to expand this area as much as possible.
Blind Spot of business needs and expectations, including requirements that the business has accumulated but hasn't shared with the technology group. This is the area that business analysts and product owners seek to shrink every day.
Hidden Tech Debt is the collection of unfinished technical tasks, undocumented code, poorly architected solutions, and other procrastinated improvements the technology team has not placed in the backlog and has not shared with the business community. This debt will be a silent tax on every future project.
Latent Debt and Needs is filled with tasks and projects that everyone is ignoring but which may grow and become more complex the longer they are neglected. These might be undiscovered bugs, tech debt forgotten due to employee turnover, customer complaints that are not heard, or future product obsolescence. This area requires constant attention to minimize.
Figure 2: An adaption of the four quadrants of the classic Johari Window that models your technology gap.
With this technology gap model in mind, let's look at some important ways to make it work for your organization.
Bring Everything in the Backlog into the Open
The goal of improving communication between the technology and business teams is to create a shared place to view, organize, and prioritize all the current and upcoming tasks and projects. You may call this master list a backlog, queue, or project portfolio. It should include every task or project of every type that might compete for resources, from software upgrades to report cleanup or debugging tasks. This increased, shared visibility for both teams will create a natural place to build a better business case for more resources or time because everyone can see the full magnitude of all requests and needs.
Best practices for dealing with this backlog effectively include basic product/project management practices such as:
- If you don't already have a master list or backlog, schedule a kick-off meeting with business stakeholders to explain how it will work, how to add requests to it, who to contact, etc.
- Ensure that the backlog is well organized and visible to the business. The actual repository could be as simple as an Excel sheet on a shared drive, but ideally it would be in a tool that everyone can access at any time.
- Organize requests into bigger projects or epic stories. This will make large lists more manageable.
- Carefully tag each request with the requestor, tech components, effort estimates, and any other metadata that would help facilitate planning and communication.
- Designate a single person to be responsible for maintaining your backlog such as a product owner, project manager, or team manager.
- Schedule a regular weekly or biweekly meeting with a representative from each business group you support to sort through prioritized requests and tasks. Allow them to see how their requests fit into the bigger picture with other departments. This grooming process should be an hour or less and should also include updates on completed tasks and revised timeline estimates.
- Send weekly or biweekly emails with a list of top priorities and completed tasks along with a link to the website or repository of the backlog.
- Reward and recognize individuals who actively contribute to shrinking the backlog.
- Give friendly but firm reminders to those who try to skirt the official process. If a finance analyst taps the shoulder of a developer to ask for a special favor, direct them to go through the front door rather than the back door.
Shrink Your Blind Spot
When teams get busy, the natural response is to ignore all distractions. However, it is important for at least one member of the team to keep their head up, maintaining regular contact with the business departments. If the blind spot becomes large enough, if will result in feelings of frustration and detachment in business departments and ultimately result in shadow analytics teams procuring their own data management and visualization tools to take care of their needs.
Best practices for shrinking your blind spot include:
- Schedule regular meetings (at least monthly) with stakeholder representatives, including business sponsors, data consumers, data science teams, and data infrastructure support groups.
- Follow an agenda that focuses most of the time on them and their needs and upcoming changes; avoid spending all the time talking about your own team's activities.
- Discourage informal, back channel requests to the technology team (e.g., the shoulder tap). These requests create an invisible time drain that will distract from and delay the prioritized tasks in your backlog.
- If discussions with business users are vague or confusing, shift your time to documenting their business processes. This will aid communication and spur ideas for missing elements in processes.
- Create a channel with business users to receive updates on their new initiatives and emerging needs. You could request to be added to their mail groups or invited to their planning or road map meetings.
- Ask for feedback at every project milestone, not just when you deliver the final technical components. In every meeting, ask for feedback about how your stakeholders are feeling about project progress, communication, and any other concerns they might have.
Shine a Light on Tech Debt
Technical debt is unique in that it is easy to overlook and is often glimpsed in fleeting moments, then forgotten. It is like a forgotten sum of money that you owe to a friend that only comes up when you have to ask for the next favor from that friend -- you will have to pay back the debt before you can move ahead. Your goal is to bring all that debt into the backlog, even if you don't have time to address it at that moment. It will be available if you have any spare bandwidth in the future, and it can be factored into new project timelines.
Examples of technical debt that should be moved to the backlog include:
- Incomplete tasks at the end of a project. There are often tasks that were pushed aside as noncritical to go-live, such as cleaning up old data, auditing user permissions, or decommissioning old code.
- Noncritical bugs. These bugs will not stop any process but will burden any data user when they have to clean up the data. For example, you may have inconsistent data in a field that report builders must fix.
- Data set pruning. Often excessive data is included in early versions of a data warehouse or imported into a data visualization just in case it is needed in the future. This may be unused tables (e.g., importing every possible object from an API when you only need a few), unused columns (e.g., dimensions with dozens of useless attributes), or unused rows (e.g., 20 years of historical data when the business only needs three).
- Poorly architected, nonmodular components that will not extend well in the future.
- Incomplete or missing documentation.
- Lack of training and mentoring for new staff.
Explore Your Latent Debt and Needs
The word latent is defined by Lexico as "existing but not yet developed or manifest; hidden or concealed." Latent requirements are the most difficult to deal with because they are, by definition, not something anyone will naturally present to you in a predictable way in a meeting. These technology gaps are a common byproduct of high staff turnover or rushed projects. They are also a natural product of emerging business disruptions and market changes.
One key to discovering these latent gaps is to perform regular post-mortem reviews as soon as possible at the end of projects to capture gaps before they are forgotten. Employees often conduct exit interviews with an HR representative before leaving a company, but the most important exit interview should be with the technical leads on the team to capture all of the unfinished tasks and ideas before they are lost. Tech teams can also conduct periodic brainstorming sessions at least monthly to identify latent requirements as part of every sprint planning session or road map discussion. Finally, remember to reward, not punish, the disclosure of previously unknown issues.
A Final Word
The relationship between technology teams and their business analytics stakeholders requires time and energy to strengthen and will be subject to all of the challenges and disfunctions that any other type of relationship will encounter. A communication model such as the Technology Gap Window helps you focus your efforts and create a better shared understanding of priorities.
Most important, it will help you communicate the full scope of work to be done to allow for better budgeting and project/sprint planning.