We are just weeks away from the
TDWI World Conference in Chicago (May 5-10), where the theme will be “Big Data Tipping Point.” I have it on good authority that by then, the current coldness will have passed and Chicago will be basking in beautiful spring weather. (If not, as they say, wait five minutes.) The theme of the World Conference is “Big Data Tipping Point,” which means that TDWI will feature many educational sessions to help you get beyond the big data hype and learn how to apply best practices and new technologies for conquering the challenges posed by rising data volumes and increased data variety.
I would like to highlight three sessions to be held at the conference that I see as important to this objective. The first actually does not have “big data” in its description but addresses what always appears in our research as a topmost concern: data integration. In many organizations, the biggest “big data” challenge is not so much about dealing with one large source as integrating many sources and performing analytics across them. Mark Peco will be teaching
“TDWI Data Integration Principles and Practices: Creating Information Unity from Data Disparity” on Monday, May 6.
On Wednesday, May 7, Dave Wells will head up
“TDWI Business Analytics: Exploration, Experimentation, and Discovery.” For most organizations, the central focus of big data thinking is about analytics; business leaders want to anchor decisions in sound data analysis and use data science practices to uncover new insights in trends, patterns, and correlations. Yet, understanding analytics techniques how to align them with business demands remains a barrier. Dave Wells does a great job of explaining analytics, how the practices relate to business intelligence, and how to bring the practices to bear to solve business problems.
The third session I’d like to spotlight is
“Building a Business Case for Big Data in Your Data Warehouse,” taught by Krish Krishnan. A critical starting point for big data projects and determining their relationship to the existing data warehouse is building the business case. Krish is great at helping professionals get the big picture and then see where to begin, so that you don’t get intimated by the scale. He will cover building the business case, the role of data scientists, and how next-generation business intelligence fits into the big data picture.
These are just three of the many sessions to be held during the week, on topics ranging from data mining, Hadoop, and social analytics to advanced data modeling and data virtualization. I hope you can attend the Chicago TDWI World Conference!
Posted by David Stodder on April 10, 20130 comments
Happy New Year to the TDWI Community! As we head into 2013, it’s clear that organizations will continue to face unpredictable economic currents and regulatory pressures, and will require better intelligence and faster decision processes. TDWI has just published a new Best Practices Report that I wrote, “Achieving Greater Agility with Business Intelligence.” This report focuses on how organizations can develop and deploy BI, analytics, and data warehousing to improve flexibility and decision-making speed. I hope you can attend our upcoming Webinar presentation of the report, to be held on January 15, which will look in-depth at the research findings and offer best practices recommendations for increasing agility.
Three key areas of innovation in technologies and practices that I covered in the report will clearly be important as organizations aim for higher agility in 2013. These include the following:
Managed, self-service BI and analytic data discovery of structured and unstructured data: Decision makers are demanding tools that will allow them to access, analyze, profile, cleanse, transform, and share information without having to wait for IT. They will need access to more than just historical, structured data found in traditional systems. Unified access to both structured and unstructured data is growing in importance as decision makers seek to perform complete, context-rich analysis against big data.
New data warehousing and integration options, including virtualization: Data integration can be the source of challenging and expensive problems. Organizations are evaluating the range of options, including data federation and virtualization, that can give users managed self-service. These could allow users to work more iteratively with IT to create comprehensive views of data in place without having to physically extract and move it into an application, data mart, or specialized data store.
Agile development methods: The use of agile methods, now a mainstream trend in software development, is having an increasing impact on BI and data warehousing. Organizations are proving that they by implementing Scrum and other techniques, they can remove a good deal of the wait and waste of traditional development processes.
In the report, we found that most organizations regard their agility – that is, their ability to adjust to change and take advantage of emerging opportunities – and merely “average.” No doubt, organizations seeking new competitive advantages in 2013 will demand better than that. They will be looking to their BI, analytics, and data warehousing systems to help them become reach a higher level of agility.
Posted by David Stodder on January 7, 20130 comments
Business decision cycles are turning faster, and to keep up, executives and managers are in constant need of new data and new types of reporting and analysis. Dynamic organizations are demanding greater agility from their business intelligence (BI) systems. TDWI Research is currently examining how well organizations are able to adjust their BI and data warehouse (DW) development, deployment, and management to enable greater agility.
How is your organization doing in addressing user demands for more agile BI/DW? What are your toughest challenges? We would very much like to include your opinions and insights in the TDWI Research survey, which is live right now. Thank you to everyone who has already participated in the survey. As part of my research for what will ultimately be a TDWI Best Practices Report, I am also conducting interviews with professionals to understand their experiences with agile development methods for BI/DW and with deploying self-service BI, data virtualization, and other technologies that are helping organizations become more agile. If you are interested, please drop me a line at [email protected].
With survey data coming in, it’s hard not to take a peek at what we have so far. Respondents say that the business factors having the most disruptive impact, requiring greater business and IT agility, are increased competition (74%, with 20% calling it “very disruptive”), economic or global instability (68%), shorter decision cycles (65%), and technology modernization (62%). Changes in customer behavior form the fifth highest factor, with 60%. The largest percentage of respondents (45%) say that their organizations are “average” at adjusting to change and taking advantage of emerging opportunities, with 10% saying that their organization is “excellent,” 31% saying “good,” and 14% saying “poor.”
Other questions in the survey will provide data for deeper insight into where challenges are most acute in terms of BI/DW development processes and technologies. One of the biggest issues regarding agility is, of course, agile software development method adoption. Ralph Hughes, chief systems architect for Ceregenics and I will be speaking on this topic on September 20 at the upcoming TDWI World Conference in Boston. If you would like to hear a preview of what we will be talking about, including the ongoing research effort into use of agile methods, listen to our recent Webinar.
Achieving greater agility through better methods and technology is a hot area of interest in the TDWI community. Let us know your views on this important topic, both by taking the research survey and by getting in touch.
Posted by David Stodder on August 27, 20120 comments
Personal, self-service analytics and discovery is one of the most important trends not only in business intelligence (BI) but in user applications generally. Expensive, monster systems that have big footprints and are not flexible to meet dynamic business needs are increasingly viewed by users as legacy. Rather than work with monolithic, one-size-fits-all applications that are dominated by IT management and development, users today want freedom and agility. They do not want to wait weeks or months for changes; they want to tailor reporting, analysis, and data sharing to their immediate and often changing needs.
I recently wrote a TDWI Checklist Report on this topic. The report offers seven steps toward personal, self-service BI and analytics success, from taking new approaches to gathering user requirements to implementing in-memory computing, visualization, and enterprise integration. I hope you find this Checklist Report useful in your BI and analytics technology evaluations and deployments.
An important conclusion in the report is that perhaps ironically, IT data management is absolutely critical to the success of personal, self-service analytics and discovery. Nowhere is this truer than with enterprise data integration. Business users often require a mix of different types of data, including structured, detailed data, aggregate or dimensional data, and semi-structured or unstructured content. In addition, given that it is doubtful that users will give up their spreadsheets any time soon, systems must be able to import and export data and analysis artifacts to and from spreadsheets. Assembling and orchestrating access to such diverse data sources must not be left up to nontechnical users.
Thus, even as users celebrate the trend toward personal, self-service analytics and discovery, its success hinges on IT’s data management prowess to ensure data quality, enterprise integration, security, availability, and ultimately, business agility with information.
Posted by David Stodder on June 28, 20120 comments
At the recently concluded TDWI Solution Summit on big data analytics in San Diego, a discussion topic that percolated throughout the conference was the increasing role in IT purchases of the marketing function and chief marketing officers (CMOs). During a question-and-answer period, an attendee asked sponsor panel speakers for comment about a January 2012 projection by Gartner research vice president Laura McLellan that by 2017, CMOs will spend more on IT than CIOs. Though impressed by the projection, the panelists did not seem surprised by this trend.
Analytics adoption is driving major changes in marketing functions, which in most organizations are empowered with the responsibility for identifying, attracting, satisfying, and keeping customers. Marketing functions are becoming increasingly quantitative; they are replacing “gut feel” with data-driven decision making. Data drives the pursuit of efficiency and the achievement of measurable results. Marketing functions are key supporters of “data science,” which is the use of scientific methods on data to develop hypotheses and models and apply iterative, test-and-learn strategies to marketing campaigns and related initiatives.
In the new TDWI Best Practices Report I wrote, “Customer Analytics in the Age of Social Media” (to be published in early July), our survey found that in the majority of organizations (59%), IT and data management functions are still the owners of the budget for customer analytics technologies and services. TDWI did, however, discover a growing budget role played by marketing and advertising functions. Nearly two out of five (38%) respondents said that this function has responsibility for the customer analytics budget in their organizations. Executive management (39%) is also a significant player in budgetary decisions. (Note: “Big data analytics for better customer intelligence” is the theme of the next TDWI BI Executive Summit in San Diego.)
Whether located in IT or under the aegis of the corporate marketing function, specialists in customer analytics must often consult with globally distributed, departmental marketing teams as well as other business units to understand key business challenges and opportunities that should be considered in the development of models, algorithms, queries, and data files for analysis. In other words, customer analytics professionals must be able to live in both technology and business worlds and work with diverse teams from not only marketing, but also finance and operations, to develop accurate, consistent, and common metrics for evaluating results. The ability to move across functions is important for delivering holistic, or enterprise, benefits from customer analytics that go beyond marketing.
Customer analytics and the budget for analytic processes are often in the middle of tensions between IT and marketing. In interviews for this report, TDWI found that the growth in analytics implementation by marketing functions is putting stress on relations with IT over control of the data and who develops and runs analytic routines. The iterative, discovery-oriented qualities of predictive modeling and variable development don’t fit well with IT’s standard approach to gathering all user requirements at once and owning the development of a solution. “IT would ask us to identify the fields we wanted,” a marketing data analyst interviewed for the report said, “but we had to say, ‘Gee, we won’t know until we can look at what’s available and start playing with it.’”
Analytics is thus rising as a sensitive – and competitive – issue as marketing functions gain a larger share of organizations’ technology budgets. It is imperative, therefore, that CMOs and CIOs communicate effectively about shared customer analytics objectives to avoid letting internal budgetary battles become an obstacle to business success. Functioning in a complementary and collaborative fashion, marketing and IT functions can achieve more together than either could accomplish alone.
Posted by David Stodder on June 18, 20120 comments
These days have been a whirlwind of projects. One of the biggest for me is the TDWI Best Practices Report I am working on, entitled “Customer Analytics in the Age of Social Media.” This report looks at what organizations are doing and could be doing to analyze information sources to improve their knowledge of and engagement with customers. Social media data is the revolutionary force in this realm; marketing functions are highly focused on how to take advantage social media both as a new channel and as a critical source of information about customer and market behavior. The heart of this report will be about how customer intelligence and analytics efforts are being reshaped by the influence of social media. This is exciting stuff.
When people talk about “big data,” much of the time they are talking about data generated by human behavior in social networks, blogs, chat rooms, comment fields, and more. Indeed, this can amount to a fast-moving, highly diverse “tsunami” of data that includes both internal (e.g., contact center interactions) and external sources. By discovering insights from this information, organizations can broaden and deepen their understanding of customers and get closer to a 360-degree view.
In addition, organizations can use social media data to gain an early view of the efficacy of marketing campaigns and product introductions. Many organizations are “listening” to such reactions in social media; leading organizations analyze the data rapidly and move quickly to adjust campaigns and engage in the social conversations to improve results.
To be sure, some organizations have serious reservations about social media data. First, not all organizations I have spoken with for the report find social media data to be trustworthy and take such analysis with a heavy grain of salt. My research found that while “gut feel” is losing out to the power of data analysis in most marketing functions, there’s still healthy debate about the real value of social media data to marketing decisions.
Second, while organizations at the leading edge of social media get a lot of attention, in a broad sense we are still in the early days. In our research, just 26 percent of participants said that their organizations are currently analyzing social media data; 22 percent are planning to do so within one year, while 21 percent have no plans to do so.
Where I found that organizations are gaining huge value is in drawing insights from social media to help them get closer to a 360-degree view of customer activity. Data silos are a problem in marketing; each channel often has its own dedicated applications and data. If organizations can correlate what they are seeing in social media with performance data from Web sites and other channels, they can begin to connect the dots across channels.
“Social media for us is not one isolated channel,” a data analyst at a large advertising services firm told me. “We use social media to gain an integrated view of the impact of our marketing across all of our channels, including billboards.” His organization is comparing social media data with their sources on marketing spending, customer transactions by location, and Web site performance. While not complete by itself, social media activity analysis enables a far more current view of marketing campaign performance than organizations have previously had.
“To see and be seen” is the credo of social media engagement. It isn’t enough to just listen; organizations have to be prepared to act. To do so intelligently, however, organizations must use social media data as not just a single source but as part of their integrated view of customer information.
Posted by David Stodder on April 12, 20120 comments