TDWI Anaheim Leadership Summit

CASE STUDY: A Case Study Comparing Continuous Experiments Versus Classic A/B Tests

August 7, 2018

Jeremy Gu

Jeremy W. Gu

Senior Data Scientist


At Uber, we use data science for everything from predicting rider ETAs and building better maps to determining optimal driver-rider trip pairings and even improving customer engagement through email campaigns. In addition to sharing updates, these email campaigns are used to inform customers about services, specials, and opportunities to engage with our services that they may have not otherwise considered, thereby improving the overall customer experience.

The Uber Eats Customer Relationship Management (CRM) team for the EMEA region launched an email campaign to encourage order momentum early in the customer life cycle. The experimenters planned to run a campaign with 10 different email subject lines and to find the best subject line in terms of the open rate. In this talk, we will introduce an increasingly popular experimental method called multi-armed bandit experiments (MAB).

How do we determine how many users should be sent to each subject line group while the experiment is running? Previously, we ran A/B tests to find out the most effective subject lines. During this campaign, we adapted to the intermediate results of our tests, optimizing the overall email open rates while analyzing data from the experiment. Classic A/B tests are not designed to solve such optimization problems.

Register Online

Rest easy—online registrations for this conference are secure. Our secured server environment keeps your information private.

Subscribe to Receive conference updates via email

TDWI Anaheim Leadership Summit

Disneyland Hotel
Anaheim, CA
August 6–7

As to Disney properties/artwork: © Disney


  • Download on the App Store
  • Get it on Google Play