CASE STUDY - High-Volume, Real-Time Data Transformations Become Reality at Overstock.com
Commentary by Jack Garzella, Vice President of IT Operations, Data Warehousing, Reporting, & Analytics, Overstock.com, Inc.
Company Overview
Overstock.com, Inc., is an online “closeout” retailer offering discount, brand-name merchandise for sale over the Internet. The company offers its customers an opportunity to shop for bargains conveniently, while offering its suppliers an alternative inventory liquidation distribution channel.
Managing the Data Deluge
With several terabytes of data stored intheir enterprise data warehouse, and millionsof transactions taking place each day,Overstock was constantly battling to keepthe data in its data warehouse accurateand up to date.
To better track and improve e-mail campaigns, and to better understand and manage customers and suppliers, Overstock wanted to establish an IT environment that would allow management to understand at any point in time how well it was meeting its performance target metrics.
Having access to keybusiness metrics inreal time is no longera fantasy.
Jack Garzella, Overstock.com
To gain access to relevant, timely information—information that could be used to help guide business decisions—Overstock realized that it needed to integrate data from disparate data systems and applications. Thus, they set out to find a data integration product that could handle the company’s high-volume, highly complex loading and transformation requirements in near real time.
Meeting the High-Performance, High-Volume Data Integration Challenge
After looking at various products, Overstockselected Sunopsis’ Data Conductor.Several factors influenced their choice:
First, Overstock really liked theperformance gains provided by Data Conductor’sclean architecture and approachto moving data. “It is not a typical ETL[extract, transform, load] tool,” said JackGarzella, who runs Overstock’s IT operations,data warehousing, reporting, andanalytics. “With a traditional ETL tool,we would have been forced to rely upon aproprietary engine to perform our transformations.We didn’t like the idea ofhaving to move our data first out of thedatabase, then into the proprietary enginefor processing, and then back into thedatabase. This approach—because of themultiple data movements and the largevolumes of data we deal with—wouldhave made it nearly impossible for us togain access to data in near real time.”
Overstock realized that by choosing atraditional ETL product, they would notonly experience performance issues due tothe way the proprietary engine handleddata, but they would also be forced toincur additional hardware—adding furthercosts and complexity to their ITenvironment. “With Sunopsis’ uniqueE-LT [extract–load, transform] approach,we are able to use the power of our existingRDBMS engine to perform the transformations,”Garzella said. Data Conductor’suse of business rules to handle integrationprocesses, and the product’s ability tohandle both batch and real-time integration,were other key factors influencingOverstock’s decision to choose Sunopsis.
Simplicity, Performance,and Scalability
Historically, Overstock handled transformationrequirements by writing customcode anytime data needed to be modifiedor moved. This was a time-consumingand tedious task, since the company processesmillions of records per day. Withdata stored and written in multiple formatsand files, and with each transactionon the Overstock.com site requiring datato be populated in six to eight differentrecords, data movement was a constantchallenge.
“Upon looking at Sunopsis, we realizedthey were a perfect solution for meetingour high-volume, high-performancerequirements, complex transformations,and near-real-time needs,” saidGarzella. Using Sunopsis’ highly powerfulETL product allows Overstock toautomatically and efficiently load andtransform large volumes of data from sourcesystems directly into their enterprisedata warehouse.
Sunopsis—Better Data, BetterIntelligence, Better Decisions
Sunopsis Data Conductor’s unique architecturemakes it possible for Overstockto automatically transform over five milliondaily transactions into useful data.Using the product, Overstock is ableto perform even the most complex datatransformations on very large volumesof data without experiencing any performanceproblems.
“Having access to key business metricsin real time is no longer a fantasy,”said Garzella. “With Sunopsis, over 300Overstock users in the sales, finance, marketing,and merchandising teams are nowable to view their relevant data in near realtime hourly, daily, or weekly, dependingupon their needs—making it easier tomake better decisions and better managethe company’s bottom line.”
This article originally appeared in the issue of .