Q&A: How Strategic Planning Can Help Manage Big Data
BI and analytics expert David Loshin discusses how strategic planning can help you manage new ideas and technologies, including big data.
- By Linda L. Briggs
- January 29, 2013
Are organizations prepared to pilot new techniques around big data, then absorb those methods back into the existing infrastructure? In this interview, BI and analytics expert David Loshin discusses ways that enterprises can successfully adjust to big data -- or any new tool or technology.
Loshin is president of Knowledge Integrity and a recognized expert in information management, information quality, business intelligence, and metadata management -- topics about which he writes and speaks frequently. His latest book, The Practitioner's Guide to Data Quality Improvement, looks at practical aspects of data quality management and control.
BI This Week: Are we as an industry rushing too quickly to embrace big data as a new source of information?
David Loshin: I am not sure that I would say that we are rushing to embrace big data. Rather, I might suggest that what we are classifying as "big data" analytics and management today reflects evolutionary steps applied to methods and techniques that have been in development over a relatively long period of time. There is at least a 20-25 year track record of some organizations taking advantage of parallel and distributed computing. It's only now, with commodity hardware and software tools generally available at a low barrier to entry, that makes what we are calling "big data" somewhat attractive and appealing.
What I question is whether organizations are properly prepared to both pilot and try out new techniques, and then absorb those methods back into the existing infrastructure. It's one thing to download Hadoop and build the environment, and even load the data or run some analyses. It's another thing to be able to leverage the existing business intelligence and analytics architectures and frameworks and meld the old with the new to enable business customers to exploit actionable knowledge.
Why do we need new types of analysis for big data instead of what we're already using?
Interestingly, a large part of what is being done in the big data world is not really new; organizations using these technologies are often looking to improve the performance of analyses they are already doing. However, they want to scale up -- to analyze bigger data sets, add more data sources, or do their analyses faster, so these types of organizations benefit from using the operating and programming models that parallelize computation, distribute data, or both.
That being said, though, there are numerous instances of companies looking to do some new types of analyses -- for example, graph analytics algorithms can answer questions that the standard relational database management system struggled with. As more companies expand their user base beyond the so-called "C-level" users into a more mixed-workload community of users, the need for scalability and performance will put pressure on the IT teams to provide high- performance solutions.
If enterprises want to introduce new tools and technologies for dealing with big data, what's the best way to do that company-wide?
A different way of asking that question would be from a more general approach -- if you want to introduce any new tool or technology, how do you engineer new technology testing, assessment, and integration, as well as abandonment, as part of the general system development life cycle?
We often have a culture of conflicts -- the IT teams want to embrace and try out new technologies but cannot get the business to buy in, so they resort to a "skunk works" approach off to the side, which drains resources without necessarily providing visible results. At the same time, business users frustrated with an apparent lack of innovation will bypass the standard processes and implement new tools via a "shadow IT" approach. Both approaches are prone to failure and confusion.
An entirely different method, and one that we have seen work, is the specific allocation of time and resources to evaluating and piloting new technologies. That means partnering with business users to try to solve real business problems, and it's done knowing full well that 90 percent of those pilots are destined for failure. However, when one approach in the mix proves to add value, at least potentially, there must be well-defined processes for technology mainstreaming that maps the architecture and integration plan for incorporating the new.
What about issues with oversight and governance -- of data, of technology, and of projects? Is the rush to embrace big data sidestepping all of that?
Of course, we want to insist on some level of governance of the process, but on the other hand, we don't want to stifle creativity by imposing too much bureaucracy. There must be a common ground in which those engaged in speculative design and development using new technologies like big data are still accountable from a programmatic standpoint, with defined deliverables and milestones, as well as some level of assurance that data quality and usability expectations are not being ignored. At the same time, as I said before, there has to be a process for mainstreaming successful pilot projects back into the "blessed" enterprise architecture. At that point, the systems would have to be realigned into all the governance hierarchies.
What is the best way to align an analytics program with its business consumers? Too often, these sorts of programs seem to be driven by IT -- why is that, and what is the correction?
This is not a new challenge. At Knowledge Integrity, the best way we've found to address this issue is using a holistic approach to aligning business value with technical initiatives. We work with the business partners to identify how their business processes can be enhanced using different approaches to analytics. The goal is to review the dependencies on information and expectations for mapping analytic results to increased profitability (either through increased revenues or decreased costs), reductions in risk, and enhancement to the customer experience.
If the costs associated with introducing new approaches to analytics can be justified in terms of potential lift, the business consumers will generally support the activity.
How can companies adhere to existing business objectives and budgets, which are often crafted well ahead of time, while still remaining agile and open to trying new technologies?
I wish there were an easy answer to this, since, as you say, budgets are often configured long in advance of the actual work being done, without a lot of expectation for absorbing new technology. We actually have guided our clients to introduce R&D budget line items specifically for researching, evaluating, and trying out new technologies, but we also caution them to ensure that these tasks are still aligned with business needs (as per the answer to your last question), as well as having the expectation that 90 percent of what is being tested will never make it to see the light of day.
In regards to new technologies and challenges, like big data management or analytics, how can organizations "plan for change" in general? What mistakes do you see clients making in this area?
I would say, try to maintain a degree of flexibility that can enable your infrastructure to embrace change instead of being resistant to it. That idea has to span people, process, and technology, by the way. One interesting way to do this might seem somewhat counterintuitive. That is, to insist on defining, adhering to, and enforcing standards: modeling standards, process standards, data standards, and so forth. One of the biggest challenges in planning for change is figuring out what needs to change.
All of this refers back to some of my earlier points: an environment that is well-governed will ultimately be much more adaptable, and that will enable flexibility and agility throughout the enterprise.