4 Recommendations for a Modern Data Ecosystem
If the idea of an ecosystem seems daunting, you're not alone. Data platforms seem easier to build and manage, but they can be difficult to change when you need to adapt to new technologies.
- By David Stodder
- April 11, 2017
A data platform sounds like something that won't change very often, if ever. Many organizations had that fixed state in mind when they installed their enterprise database and data warehousing systems. These systems took a long time to build and put into production, so (except for the occasional upgrade or tuneup) they should last a long time -- maybe forever -- right?
Not quite. Tumultuous changes are underway in business intelligence (BI), analytics, and data warehousing, pushing organizations to take a new perspective on their data platforms. In fact, rather than "platform," some experts now use the term "ecosystem" to describe the emerging data environment. The term is an acknowledgement that going forward, organizations will need to focus on the integration and interdependence of multiple platforms.
[Editor's note: TDWI's upcoming Chicago Conference and Leadership Summit (May 7-12) will focus on the modern data ecosystem; educational sessions, case studies, panels, and informal group discussions will examine such components as big data, data science, self-service BI, analytics, and new approaches to data integration and preparation.]
In my research and discussions with technology leaders, I am seeing several trends that are influencing the direction of the modern data ecosystem. Following are four recommendations based on those trends.
Recommendation #1: Make Governance a Priority
Governance is critical to the healthy operation of any organization. Without it, business users risk losing trust in the data and the resulting analytics; the organization can also be exposed to regulatory violations. As an organization's data ecosystem grows more diverse, governance becomes an even greater priority because IT may no longer have clear oversight of the data and how it is being used.
Thus, it's not surprising that governance is a major topic in the industry. Technology solution providers are aware of the interest in governance and are starting to provide better tooling for monitoring data both at rest in databases and in motion across the organization.
Data preparation technologies are important to data governance because they can help organizations build a better knowledge base about the data, how data is related in multiple sources, and how it is being transformed for BI and analytics. This is important for understanding data lineage, which is critical to enabling users to trust their BI and analytics. With data lineage, organizations can understand where the data originated and track how it was changed and transformed.
Recommendation #2: Prepare to Democratize Data Science
Just as self-service BI and visual analytics have expanded to become "democratized" in organizations, data science is also becoming a more mainstream activity. In coming years, organizations will become less dependent on "unicorns" -- rare individuals who are part statistician, part programmer, part data analyst, and amazingly knowledgeable about the business domain. Unable to find and keep that one person, many organizations have been smartly assembling data science teams where experts in each of these areas work together.
Some in the industry bristle at the term "citizen data scientist," but it is one way to describe a growing constituency of business users who are doing more than self-service BI and visual analytics but are not full-fledged data scientists. They will build and test analytics models, perform statistical analysis, and employ machine learning features embedded in next-generation tools and workbenches.
Organizations need to assemble their data ecosystem with a strategy for supporting more widespread data science, including providing access to data -- in data lakes, cloud-based platforms, or in-memory computing close to the users -- so that many users can perform data science activities.
Recommendation #3: Build and Share Customer Intelligence
In every industry today, businesses feel a fierce urgency to become customer-centric. They want to preserve and expand existing customer relationships and attract the best new customers. With new analytics insights, companies seek to streamline and improve their operations and develop innovative products and services.
Thus, it's not surprising that the latest techniques in data visualization, big data analytics, artificial intelligence, and machine learning are being used to improve customer intelligence and apply it to operational decisions.
Organizations need to ensure that their data ecosystems have the right mix of technologies to support continuous pursuit of customer insight. Marketing, sales, and service functions will be the main stakeholders, but corporate leadership and product development teams are also increasingly demanding a steady diet of knowledge about customer trends. Users will need to tailor their views of customer intelligence to fit their business context and concept of actionable information.
Recommendation #4: Create a Hybrid, Flexible, and Open Architecture
With enterprise data warehouses and big data lakes, on-premises systems and cloud-based systems (including platform- and software-as-a-service), and historical data and real-time streaming data, organizations have to avoid information architecture that is too rigid.
Business users working on analytics as well as new data-driven applications are likely to draw on several of these sources to satisfy information demands, which will put pressure on data integration and the quality of metadata and master data resources. Organizations will also need the ability to direct workloads to the right platforms to take advantage of their respective capabilities.
Drawing from the Community
If the idea of an ecosystem seems daunting, you're not alone. Platforms (not to mention monolithic legacy systems of tightly integrated technologies) seem easier to build and manage. However, platforms can be difficult to change when you need to add new technologies and respond to new types of users and demands. An ecosystem, by definition, is expected to evolve.
The TDWI Conference and Leadership Summit will be a gathering of peers who are seeking to learn strategies for moving data resources toward an ecosystem. TDWI events bring together professionals from a variety of industries, so attendees will have a chance to exchange ideas and relate experiences from different perspectives. I hope you can come to Chicago and take the opportunity to learn from both industry experts and case studies by organizations that have made progress with these emerging practices.
About the Author
David Stodder is director of TDWI Research for business intelligence. He focuses on providing research-based insight and best practices for organizations implementing BI, analytics, performance management, data discovery, data visualization, and related technologies and methods. He is the author of TDWI Best Practices Reports on mobile BI and customer analytics in the age of social media, as well as TDWI Checklist Reports on data discovery and information management. He has chaired TDWI conferences on BI agility and big data analytics. Stodder has provided thought leadership on BI, information management, and IT management for over two decades. He has served as vice president and research director with Ventana Research, and he was the founding chief editor of Intelligent Enterprise, where he served as editorial director for nine years.