TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

The ETL Market Reborn

The ETL market had a surprisingly successful 2004—and is poised for even more growth through 2009

• By Stephen Swoyer
• December 7, 2005

If you ever thought ETL was a dead or dying market, IBM Corp.’s $1.1 billion acquisition of the former Ascential Software Corp. earlier this year should’ve been enough to convince you otherwise. You don’t, after all, pony up that kind of cash for a company unless you believe its bread-and-butter technology has legs.

Now there’s new research from market watcher Gartner Inc. that says the ETL market had a surprisingly successful 2004. According to Gartner’s data, the worldwide ETL tools market grew by 6.7 percent last year. It isn’t runaway growth, but it’s encouraging nonetheless. What’s more, Gartner projects an ETL market compound annual growth rate of 6.3 percent through 2009, which should help generate half a billion dollars in new license revenue.

But this news isn’t new. In a report dated April 25, 2005, Forrester Research stated that “ETL grew 13.5% to $949 million in 2004 and will pass the $1 billion milestone in 2005.” The report – titled “Demand Is Growing For Traditional And New Uses Of ETL” – explains that growth is driven by “expanding user practices, whether traditional (data warehousing) or new (general data integration outside warehousing).”

There’s a further wrinkle, however: ETL as we know it is fast disappearing. “This market is rapidly evolving into a market of multipurpose data integration platforms suitable for use beyond the traditional domain of … BI and data warehousing,” write Gartner analysts Colleen Graham and Ted Friedman. “As the market continues to evolve, larger vendors are building out their products toward this end, putting increased pressure on small pure-play ETL vendors and resulting in additional market consolidation. As a result, ETL, as a market, will not exist in the long term; instead, ETL will be one integration pattern among many that is supported by the new breed of data integration platforms.”

“The trend with vendor tools is to include capabilities drawn form other types integration tools,” said Philip Russom, senior manager of research and services at TDWI. “For instance, EAI tools support more data transformations, data quality tools can now integrate data, EII tools operate in real time like EAI, and so on. ETL tools have seen a really dramatic evolution this decade, in that most now embed substantial capabilities for messaging, data quality, profiling, and EII. Although ETL will always be at the heart of these tools, we’re quickly moving toward what I’ve called a ‘Universal Integration Platform’ (UIP). The advantage of the UIP for users is that they can get best of breed ETL – plus closely related capabilities – in a single tool, deployed as a single server. If ETL is to escape its data warehouse ghetto, it’ll have to develop the broad appeal of the UIP.”

According to Forrester, ETL is well on its way into new markets. “Data warehousing is still the bread and butter of ETL.” Today, data warehousing is 80% of the ETL market, although the non-data-warehouse slice of this pie will grow to 30% by the end of 2007. “Note that this is a new market for ETL; in other words, non-data-warehouse usage does not scavenge from data warehouse usage, but instead complements it.”

It’s difficult to speak of winners and losers in a growing market, but -- as far as an ETL pecking order is concerned -- Gartner once again named Informatica Corp. ETL market champ -- for the fourth year running. The research giant says Informatica controlled a quarter of the overall ETL market in 2004 and at the same time grew its revenues by 10.8 percent over its 2003 performance. IBM/Ascential, on the other hand, was the number two player, trailing market leader Informatica by just over 5 percent and growing its revenues by 6.5 percent. SAS Institute Inc. was third, overall, but posted the most impressive growth -- 21.2 percent in 2004. Together, the big three ETL vendors controlled 60 percent of the overall market pie, with the remainder contested by a large number of small vendors, including DataMirror Corp. and Pervasive Software.

Forrester Research concurs, stating that “Ascential Software, Informatica, and SAS Institute are still the market leaders, accounting for 60% of the market in 2004. Growth comes primarily from these three vendors.”

The Gartner analysts think a number of trends will fundamentally alter the ETL marketscape, starting first and foremost with a decoupling of ETL from its BI roots. “Activity around ETL technology will continue to shift from business intelligence-oriented deployments to non-BI scenarios, such as data consistency between operational applications, master data management and system migrations,” Graham and Friedman write. “As a result, functionality such as data quality, the ability to deploy data integration functionality as a service, support for real-time data sources and seamless deployment across multiple platforms will become more important.”

Separately, competition from relational database vendors such as Microsoft Corp. and Oracle Corp. will start to take a bite out of enterprise ETL license revenues. “These vendors offer ETL functionality bundled into the DBMS or other products for free or nearly free. This will result in a significant uptake of their offerings and will limit revenue growth for the stand-alone ETL market as a whole, and it will stunt the growth of some of today's pure-play ETL vendors.”

And just as BI vendors are pushing BI tools consolidation, so, too, are ETL vendors poised to profit from a similar round of consolidation in the data integration space. The twist, of course, is that Microsoft and Oracle will almost certainly profit, too. “As organizations are increasing purchases of data integration technologies, many are standardizing on a single tool or set of tools. This will continue to drive the bulk of the market's growth to the larger ETL and data integration vendors, including the larger DBMS vendors,” Graham and Friedman argue.

By this point, you may be wondering why you should care about the ETL market. Forrester Research states the answer succinctly: “At least 75,000 user organizations worldwide rely heavily on extract, transform, and load (ETL) tools in critical business functions, so maintaining vendor robustness and overall growth in the market is important… Given two years of high growth in a row and the imminent $1 billion milestone coming in 2005, ETL is a safe bet as a long-term technology.”