Using Methods to Avoid Madness: How Agile Aids Self-Service BI and Analytics
Three recommendations based on TDWI research for applying agile methods.
- By David Stodder
- March 31, 2016
Agility -- the ability to sense change, adjust behavior, and take advantage of unexpected opportunities -- is a highly desirable quality for organizations in every industry. Firms in healthcare, financial services, hospitality, and now even taxi cab services are undergoing rapid change. Decision makers need data insights to help them be aware of changes in their organizations’ environments so they can adjust strategies and position resources to take advantage of events or circumstances rather than be left behind.
Unfortunately, too many organizations are dependent on slow and inflexible spreadsheet applications, data warehousing systems, and business intelligence (BI) application development processes. These are obstacles to business agility.
Self-service BI, visual analytics, search, and data discovery tools are exciting to business users because they offer the possibility of deriving data insights faster and with greater flexibility and relevance in terms of how users want to blend data for analysis and visualize results. Although the tools do require training, users of these newer applications and cloud-based services are less dependent on IT for data access and the development of dashboards or other visualizations. These technologies can help organizations make data-informed decisions in response to events or changing business circumstances.
However, many organizations are concerned about the “madness”: that is, the data chaos that could ensue as self-service spreads and users take more control over their data and analytics. IT functions have resisted the self-service trend for a variety of often good reasons, including concerns about data security, data quality, governance, and whether their data systems are prepared for rising performance and availability demands.
However, more IT leaders are realizing that there’s little to gain from fighting such a popular trend and that there are benefits to giving users more control. Users can explore and experiment more on their own instead of sitting in the backlog and grousing about how they can’t get IT developers’ time.
Project complexity grows once organizations step beyond deploying self-service technologies merely to provide users with better front-end visualization tools for existing spreadsheets and other well-defined data. Both new and familiar BI and data warehousing challenges crop up. Users need more varied data access, more frequent data updates, and support for mobility. Many want new types of functionality that integrate visualizations, data changes, and analytics insights more tightly with their other business application rules and processes. Some would like to enable visual analytics as services for their customers and business partners, which raises the bar even higher in terms of performance and availability.
Very quickly, it can become clear that if organizations do not revise their project development methods and take steps to encourage better collaboration between users and IT developers, their investment in self-service tools will deliver limited business benefits.
Agile Methods plus Self-Service Technologies
One of the quiet revolutions happening in many organizations is the adoption of agile software development methods for BI and data warehousing projects. Although some projects still require traditional “waterfall” development, agile is helping many organizations overcome the waterfall’s drawbacks, particularly in cases where it is hard to define requirements before business users have had a chance to explore the data. Self-service BI, visual analytics, and data discovery tools can help users do just that.
Agile methods promote shorter development cycles and delivery of iterations or releases that users can test and implement sooner, rather than wait for a project to reach an end state that could be months (if not years) away.
At our February TDWI Executive Summit in Las Vegas, the growing popularity of agile methods for BI and visual analytics projects involving self-service technologies was on full display. No doubt the topic will be discussed frequently in sessions and in networking opportunities at the upcoming TDWI Conference and Executive Summit in May in Chicago.
At the Summit, David Han, senior manager of business intelligence at the travel and expense software company Concur, and Ted Corbett, CEO of Seattle-based consulting firm Viztric, discussed of how Concur applied agile plus self-service BI and visual analytics to increase data-driven decision making in the company. Han and Corbett offered many best practices and key success factors. They strongly advised Summit attendees to look for opportunities for quick wins in creating business value, which was something that they found agile methods to be helpful in enabling. One critical focus was the development of global key performance indicators (KPIs); business units at the company were reporting widely different numbers for the same metrics, so this addressed a major pain point.
Han and Corbett said that Concur’s BI and visual analytics projects employed all phases of agile methods including two-week sprints and daily scrum meetings of all team participants. To keep the pace of development moving, they recommended doing light design documentation for back-end development and little or no documentation for report development, depending instead on frequent feedback with each sprint. Teams actually use BI to measure user adoption, the value being created, and various availability and productivity factors.
Han and Corbett advised attendees not to neglect continuous education of users in how to get the most out of the tools and applications. They also try to never miss an opportunity to evangelize successes to keep business leadership engaged in the projects.
Wendy Gradek, who leads EMC’s Customer Services Organization as part of its Advanced Proactive Services team, spoke at the Summit about the company’s objective of democratizing BI and analytics so that a range of users could do more with less IT involvement. The context for this effort, which under her leadership included the establishment of an internal virtual Analytics Enablement Center, was to revamp EMC’s customer services to appeal to the “digital generation” of customers who expect easier, real-time access to data about their EMC systems. Gradek’s team wanted to build data visualizations for customer services purposes that accessed EMC’s central Big Data Lake.
Her advice: organizations should implement agile processes and establish governance as they increase data democratization so customers and internal teams are empowered but also trust their data – and the IT organization can trust them to use the data properly. She also related how learning the “cultural psychology” of both customers and EMC’s internal teams was important to the success of democratization because not all users are the same. The result has been an innovative customer service approach that provides greater transparency for customers to understand what’s going on with their EMC systems.
Agile can help IT change how the function works with business users. This was a takeaway from a third Summit presentation I would like to highlight. David Schaefer, chief business intelligence architect for Intel’s sales and marketing IT group, talked about agile methods’ role in fostering a close partnership between business and IT. Agile scrums spurred his IT function to embrace the kind of changes in user expectations brought on by self-service BI and visual analytics technologies and users’ interest in accessing big data stored in Hadoop files. Schaefer quoted a director of business analytics in sales and marketing, who endorsed the agile approach by saying that “agile is now helping us increase our time-to-market velocity and really enable analytics to move at the speed of business.”
Three Recommendations
In closing, here are three recommendations based on our research for applying agile methods.
Focus agile methods on producing quality deliverables and design excellence. Although speed and flexibility get much of the attention, agile development methods can significantly improve quality. Ensure that closer collaboration between users and developers aims at quality objectives by setting up appropriate metrics.
Address changing requirements through agile method iterations. One of the key drawbacks of the waterfall method is that it usually does not address changed user requirements until after a significant amount of work has been completed, which then must be done over. In exploring data with self-service technologies, users often come up with new and different requirements. Collaboration between business users and IT developers can capture changing requirements as part of agile iteration cycles. Using self-service technologies to let users examine the data early as part of agile iterations can be a benefit.
Create small, cross-functional agile development teams rather than large ones. Our research supports experiential observations that smaller agile development teams are more successful than large ones. Agile teams should include stakeholders from all business and IT functions that have a role in the project, but many organizations have found that smaller teams integrated into a whole are better than one very large team.