The Fastest Path to Achieving Your Analytics Goals

October 16–18, 2017 | Seattle-Bellevue, WA

Trustworthy A/B Tests: Lessons and Pitfalls in Online Controlled Experiments

October 18, 2017

Prerequisite: None

Ronny Kohavi

Distinguished Engineer, General Manager, Analysis and Experimentation


The Internet provides developers of connected software, including web sites and applications, an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using trustworthy controlled experiments (e.g., A/B tests, randomized controlled trials, field tests, bucket tests, flights).

While the theory of a controlled experiment is simple, and dates to Sir Ronald A. Fisher’s experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s, the deployment and mining of online controlled experiments at scale (e.g., hundreds of experiments run every day at Microsoft) and deployment of online controlled experiments across dozens of web sites and applications has taught us many practical lessons. We provide an introduction, share real examples, pitfalls, and discuss scale challenges.

Want to hear more on A/B testing from Ronny? Check out his latest article in Harvard Business Review:

Register Online

Rest easy—online registrations for this conference are secure. Our secured server environment keeps your information private.

Gain Instant Access to Sample Accelerate Sessions

Analytics Accelerator Award

We are no longer accepting applications. Award Ceremony taking place Monday, October 16th

Sign Up for Updates!

TDWI Accelerate Seattle

Hyatt Regency Bellevue
Bellevue, WA
October 16–18


  • Download on the App Store
  • Get it on Google Play