As BI and analytics become more mainstream, organizations are realizing that it makes sense to both enrich and augment their data in order to gain more insight. Successful companies realize that utilizing traditional structured data only for analytics is a non-starter. Organizations are more often adding ‘new’ data sources to the mix, including demographic data, text data, and geospatial data to their data sets. They are also looking for external data, such as social media data, weather data, and other third-party sources. The demand from data consumers has also driven many new organizations to pursue sharing their data. Many of these data sources are cloud-based.
Fern Halper, Ph.D.
One of the strongest trends in data management today and into the future is the development of complex, multi-platform architectures that generate and integrate an eclectic mix of old and new data, in every structure imaginable, traveling in time frames from batch to real time. The data comes from legacy, mainstream enterprise, Web, and third-party systems, which may be home grown, vendor built, open source, or a mix of these. More sources are coming online from machines, social media, and the Internet of Things. These data environments are hybrid and diverse in the extreme, hence the name hybrid data ecosystems (HDEs).
Philip Russom, Ph.D.
Want to become a data engineer but aren’t sure which technologies are the right fit for the job? People switching into big data are faced with a difficult decision—should you learn MapReduce or Spark? The answer seems simple, but requires more information and insight. Answering this and other questions correctly places you on the path to becoming a data engineer.
A perfect storm of data management trends is converging.
First, organizations across many industries are experiencing the big data phenomenon, which forces them to capture and leverage data from new sources, in structures and velocities that are new to them, in unprecedented volumes. Second, technical users are scrambling to learn new data platforms like Hadoop and their evolving best practices. Third, the data lake arose suddenly in 2016 as the preferred approach to managing very large repositories of raw source data. Fourth, business managers have attained a new level of sophistication in their use big data for business value and organizational advantage.
Philip Russom, Ph.D.
Arcadia Data, HVR, Paxata, Panoply
The data universe has changed. Big data, cloud computing, and open source have dramatically expanded the number of data warehousing offerings available to today’s businesses. An increasing number of companies are implementing self-service business intelligence (BI) and visual analytics tools to access and make sense of all of the new and diverse sources of data their teams are consuming. Data literacy is changing equally fast as an increasing number of “data consumers” want to interact with data on their own rather than through IT.
Many organizations today are struggling to get value from their data and advanced analytics initiatives. The struggle begins with data diversity, as organizations are trying to support new apps, customer channels, sensors, and social media outlets. Each source may have its own data structure, quality, and container (in the form of files, documents, messages). The struggle is exacerbated by the exploding volume of data that must be captured, processed, stored, and delivered to the right users in a state that is fit for their own individual needs.
Philip Russom, Ph.D.
Certainly, every business leader wants to have trusted, secure, consistent and usable information. But data volumes and systems complexity has been increasing for years and most organizations rarely prioritize data governance, so why care now? We’re at the brink of a perfect storm of unprecedented IT megatrends. The convergence of Cloud, Social, Mobile and Big Data foreshadows the upcoming tsunami of data ripe with potential business value. But it will also make the frustrating complexity of your traditional on-premises transactional data management challenges appear amazingly “manageable” in contrast.
Claudia Imhoff, Ph.D.
TDWI and IBM Content