TDWI Checklist Report | Becoming Big Data Ready: Evolving Best Practices in Data Management
November 4, 2015
This is an exciting time to work in data management because the arrival of big data has brought with it numerous new data types, new data platforms, and new types of data consumers. In response, many organizations and data professionals are evolving their data management practices and tool portfolios to fully embrace and leverage new opportunities in data discovery, advanced analytics, and other data-driven applications.
New big data sources. The wave of new big data sources washing over many organizations brings new data-driven opportunities for modern businesses that are willing to capture and leverage big data. For example, streams from new sensors, devices, and machines enable firms to monitor and run the business in more rapid, nimble, and granular ways. Data from new smartphone apps or social media opens up new channels for managing customer relations and analyzing customer behavior. Hence, with new big data sources come new insights into customers, partners, and internal business processes that couldn’t be analyzed in the past. However, leveraging new data for organizational advantage is technically challenging because many new sources yield data that is schema free, multi-structured, or streaming. Most data professionals need to augment their skills and tools to get full business value from new big data sources.
New data analytics. The consensus among business leaders today is that big data has potential business value—if it can be curated and analyzed properly. Most believe that the primary path to competitive advantage from big data is through various forms of analytics, from discovery-oriented analytics (led by new roles such as data scientists) to more operational and repeatable reporting by traditional BI users. Early adopters already have analytic applications in place that extend older analytics for improved fraud detection; improved operational efficiencies; improved customer engagement (big data provides massive samples for these); and new analytics for social sentiment and customer behavior (in new channels such as Web, mobile, and social channels). The new analytics is worth pursuing, but its success requires new skills and tools for data management.
New data management. Organizations are facing a dual challenge: handling the growing supply of new big data and managing new data in a way that provisions data for users’ growing demand for analytics. The dual challenge cannot be met unless big data is captured, managed, and processed in ways that curate big data for timely and trusted analytics without diluting its meaning or analytical value. Hence, data integration and other data management disciplines have even more importance in the new world of big data analytics, and they are evolving to meet the needs of the modern data-driven enterprise.
Old and new are coexisting. Today, data management is satisfying the new requirements by introducing new terms and concepts such as early data ingestion, on-the-fly data processing, agile DI, proactive data quality, and data preparation (or data blending, wrangling, and munging). Even so, existing best practices need not be thrown away; they continue to fulfill traditional enterprise data requirements as they are extended or complemented by new functionality and practices.
To help the reader understand the ongoing evolution of data management—in response to new requirements around big data and analytics—this report provides a checklist of data management practice areas that are most affected. In addition, the report relates new practices to existing ones, indicating when to use which one.