Software Enables Complex Customer Searches Across Huge Database
New data warehousing and analytic software has made a world of difference for a firm that collects and rents e-mail names.
- By Linda L. Briggs
- March 17, 2010
How often does a company attribute a 300 percent growth in sales solely to its new business intelligence and data warehousing software?
At Database Emailer, that's exactly what president and CEO Al Inga says that new data warehousing and analytic software has done for his company. "In 10 months, we've gone from being an also-ran to dominating the entire industry," Inga says. "It's all because of the software solution."
Inga's company, based in Little Falls, New Jersey, owns the largest opt-in e-mail database in the U.S. The company collects opt-in e-mail addresses (people have explicitly agreed to receive product offers by e-mail, giving the names more value) and rents subsets of the database to companies that want to reach a specific demographic.
Beginning in 2008, the company began offering a service to customers that allowed them to search Database Emailer's database before a purchase to determine how many names were available in a given demographic. In a simple search of the company's hundreds of millions of names, a potential buyer might ask for a search of all females in the database living in a given area and earning within a specific income range. Often, potential buyers specify dozens of criteria, making the searches complex and challenging, especially given the immense size of the database. "People want access to information quickly," Inga says, "and they want lots and lots of criteria to choose from."
Compounding the challenge, each customer query had to be built and conducted by a Database Emailer employee. The customer would specify a set of search criteria, and a technician would create and run the search. A single query was often not enough. After a basic search showed the number of available names, the customer might ask for a refined or expanded search, changing a few variables and sending the technician back to the keyboard. That back-and-forth might continue several times. "Meanwhile," Inga says, "[the customer] is shopping all over with other data suppliers to get his order filled."
To service customer queries, Database Emailer maintained a small IT support staff sprinkled around the globe, from the U.S. to India to the Czech Republic, to respond in a timely manner to customer requests anywhere in the world. At the time, Inga says, the company was averaging 100 customer queries a day.
With over 150 million names at the time and constant complex queries underway, the company had vastly outgrown its original MySQL database. Queries involving refined searches of broad categories, such as all of North America, often crashed the system. Partitioning the database into 50 U.S. states plus territories and Canada helped, but not enough -- a search of one of the larger states could still trigger a crash. "I went to MySQL and said, 'This is crazy. We have to look into a better solution,'" Inga says bluntly.
After looking at a number of possibilities, including big-name solutions such as Oracle, Inga heeded the recommendation of MySQL and contacted Infobright, a small open source vendor that offers a MySQL-integrated analytic data warehouse. Infobright, a MySQL certified partner, uses a columnar database design, enabling more efficient data compression and better performance in many cases.
Infobright, founded in 2005 and headquartered in Toronto (though much of its development is done in Warsaw, Poland), is a commercial provider of column-oriented relational database software. The company's database software is integrated with the popular open source relational database management software MySQL. However, Infobright maintains its own proprietary data storage and query optimization layers.
With Infobright Enterprise Edition now in place, Database Emailer has seen dramatic improvements in what it can offer customers. In less than a year, the company has doubled the size of its database to over 300 million records and it continues to grow. Even with a database twice as large as in the past, gone are the days when broad searches brought the system to its knees.
Just as important, by creating a simple SQL-based interface for queries, the company now gives customers the ability to conduct their own searches through an online interface around the clock. Customer searches can take between five and 15 minutes, depending on the size of the file and the search. "That couldn't have been done before we had Infobright," Inga says.
The new system has eliminated the need for an IT person to build and conduct custom searches; allowing the company to use its staff for other purposes. "That's the kicker," Inga says. "I don't have to pay high-placed IT guys to run back and forth doing queries. Instead, they can go into marketing. … The IT guys are now salesmen. … They're making triple what they were before."
Infobright bills its data warehouse as "self-managing" because the database administrator does not need to create and maintain indexes.
Inga was pleased with the Infobright solution from the start for several reasons. "Infobright seems to not only be more scalable, but they seem to have a better solution" than any of the other products he considered, he says. In addition to the cost savings -- desperate for a solution, Inga was prepared to spend as much as necessary to improve performance -- he says he appreciates working with "a young, aggressive company that is new in the market, because you get a certain [high] level of service."
With the new system in place, his overhead is so low that Inga says he can compete with the largest companies in the industry, such as Dun & Bradstreet, in offering targeted e-mail names for much less. "We don't have any of that [overhead cost]. We are now selling data at 20 to 30 times less than [the competition] can sell it for."
In addition, Web site query times have been dramatically reduced, from an average of seven minutes to query the database and get a response, to an average of just 10 seconds now. In addition, Inga says, he is saving some 25 employee-hours a day -- time that IT staff used to spend helping customers query the database. The additional capabilities have also more than doubled the customer acquisition rate -- the company added 400 new customers between March and September 2009, double the number added in the previous six months.
According to Inga, the new system also makes it far easier to up-sell clients on more complex and more frequent searches. "[Individual] sales were bigger once we had this solution in place," he confirms. Customers can now do things themselves, he says, that "would have taken hours previously, one e-mail message [to IT tech support] at a time."
Infobright uses software that combines a column-oriented database design with the company's own architecture to help simplify managing and analyzing large volumes of data. With a columnar database design, each column in the database stores a single data type (with a row-based design, each row typically contain several data types). The column design allows more efficient data compression and can reduce hardware needs -- Infobright Enterprise Edition can scale to 50TB on a single server and provides compression from 10:1 to over 40:1, according to the vendor. Database Emailer is using the same server it used previously (with a 500G hard drive and 8G of RAM), but with the more efficient Infobright software in place, it's getting far different results.
Inga is delighted with the results he has seen and doesn't hesitate to attribute it all to the software. "Within one month [of installation]," he says, "the whole thing was paid for. Within 10 months, when everyone else's business was decreasing, ours was tripling. While the industry went down 50 percent [due to the recession], we tripled our business, and in the worst economy in history."
"The queries themselves haven't changed," Inga concludes. "The genius is in the database design. … I just hope all my competitors don't figure out how I'm doing this."