The Importance of the Universal Semantic Layer in Modern Data Analytics and BI
What is a semantic layer and how can a universal semantic layer enable your business users to deliver actionable insights and make data-driven decisions faster?
- By Ankit Khandelwal
- July 13, 2023
The semantic layer is a concept that has been around for decades. It remained an unsung hero relegated to the backdrop until recently, receiving a significant uptick in interest beginning in early 2022. What is a semantic layer and why is it gaining focus as an independent entity in today’s BI and analytics stack?
What Is a Semantic Layer?
A semantic layer is a layer of abstraction that separates the physical view of data from the view seen by business users. It provides a logical view of the data that is easier to understand and work with. By providing a more business-friendly representation of data, it acts as a bridge between the raw data and the business users.
For instance, a typical sales data store would contain separate tables for billing transactions made on e-commerce sites, stores, and other channels. An enterprise would use other tables for storing product information, customer data, marketing campaign data, and so on. Business users need this data to answer questions such as “How much product sales growth was seen in Manhattan after a local digital advertising campaign?”, “What are the highest grossing product combos consumed by millennials in Los Angeles?”, or “Has the average order value of sales through apps improved from last year?”
However, finding answers requires querying, extracting, and aggregating data from multiple tables using joins, filters, or other complex operations. The user must understand the underlying data structure and complex relations between tables -- and be familiar with query languages such as SQL. Without this knowledge, business users must rely on data analysts to provide answers, creating dependencies and delays and eroding trust in insights received secondhand. Furthermore, the process hampers their ability to explore the data themselves for deeper insights and gain topical intelligence from it that is immediately actionable.
This is where a semantic layer comes into play. As in the example, business users need dimensions (e.g., time, location), metrics (e.g., average order value, growth), and aggregations (e.g., revenue), but physical data stores actually contain fields and schema. Acting as a virtual layer between the business users and the underlying data sources, a semantic layer creates a business view of the underlying data, enabling users to access and analyze it without requiring technical expertise or knowledge of its structure.
The semantic layer does the heavy lifting of abstracting the complex underlying data into familiar business terms such as sales, revenue, customer, and product, establishing a common and standardized language across teams.
Realizing the promise of semantic layers, multiple BI tools and data discovery platforms implemented semantic layers within their products over the past decades and quickly became popular among business users.
Semantics Within BI Tools and the Fall from Favor
Although semantic models worked well within BI tools, as more business functions came to be data-driven, different departments began to adopt different BI tools. A 2020 survey by 360Suite states that the average number of BI solutions used in an organization is 3.8, with 67% of respondents having access to more than one solution.
When users created their own semantic models in disparate BI tools, it led to siloed reporting, which, in turn, led to multiple versions of business logic, use of diverse metrics, and disparate interpretations of the same data within one organization. With no common representation of data in business terms, a single source of truth became difficult to achieve -- and semantic layers lost some of their sheen.
Revival as the Universal Semantic Layer
Driven by these issues, enterprises needed to find a solution that creates a single data view across disparate BI tools and business functions. Furthermore, as data volumes exploded with digitalization, organizations migrated to modern data platforms that had the capability to handle voluminous organization-wide data. This also created an opportunity to establish standardized semantics across all reporting, analytics, and visualization solutions, resulting in BI system architects reconsidering a universal semantic layer.
A universal semantic layer is implemented as a dedicated layer between data sources and all BI tools. Irrespective of the BI tool users choose, the universal semantic layer allows them to work with the same semantics and underlying data layer, leading to insights and reports that are consistent and trusted. With clear advantages over the fragmented implementation earlier, a universal semantic layer has gained center stage by delivering multiple benefits.
A universal semantic layer:
- Connects to multiple BI and data science tools and supports various a variety of query languages, providing flexibility and compatibility with disparate platforms.
- Works on top of various data sources, providing virtualization and federation capabilities and enabling business users to access data from multiple sources.
- Allows users to define complex calculations and express complex business logic to derive deeper insights from their data.
- Allows organizations to add or upgrade data sources without impacting the existing business view of the data.
- Simplifies monitoring and management of security and governance processes, along with compliance with organization's policies and regulations.
- Optimizes seamless data access and eliminates redundancy and latency.
- Empowers business users with self-service analytics to derive granular insights quickly and easily.
Looking ahead, it is becoming increasingly crucial to establish a universal semantic layer to provide business users with a consistent view of all enterprise data and to enable them to conduct rapid analysis. Creating a semantic layer on top of all data sources ensures quick access to a single source of truth, facilitating a shared understanding of dimensions and metrics across the organization. A well-designed and high-performing semantic layer empowers business users to leverage data more efficiently, delivering actionable insights and driving faster decision-making.
About the Author
Ankit Khandelwal is the senior director of engineering at Kyvos Insights. In his career, Khandelwal has gained extensive experience in leading and managing teams, architecting big data analytics solutions, and designing and developing business intelligence and data analytics solutions based on Adhoc Analytics using OLAP and data mining, Hadoop, and Spark.