Lenny's Podcast: Product | Career | Growth

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

July 27, 2023

AI Summary

5 min read

Ronny Kohavi, a leading expert on A/B testing from stints at Airbnb, Microsoft, and Amazon, shares experiences running thousands of experiments, emphasizing how even trivial changes like swapping ad lines on Bing boosted revenue 12% or opening search results in new tabs improved outcomes across companies.

Unexpected Wins from Small Changes

Small, low-effort tweaks often yield outsized results that intuition misses, such as promoting the second search ad line to the top on Bing for a $100 million annual gain without harming user metrics, or opening links in new tabs to boost engagement at Microsoft services and later Airbnb. These "gold nuggets" are rare; most gains accumulate inch-by-inch from many modest improvements, like Bing relevance team's 2% yearly lifts or Airbnb search's 6% revenue rise from 250 experiments where only 8% succeeded. Resources like goodui.org aggregate patterns from hundreds of tests (e.g., new tabs worked 3/5 times), and Microsoft’s "Rules of Thumb" paper extracts reliable patterns from thousands of experiments to inform future ideas.

Continue reading the full summary in the app — free to try.

Read Full Summary →

Free • No credit card required

Open in App

What you'll learn

1 **[00:22] Surprising A/B Test Wins**
2 **[10:35] Rarity of Big Wins and Failure Rates**
3 **[15:29] Reusable Patterns and Institutional Memory**
4 **[20:45] Test Everything, Balance Incremental vs. Big Bets**
5 **[24:48] When to Start Testing and Scale Thresholds**
6 **[28:00] Defining OEC for Long-Term Value**
7 **[36:31] Common Pitfalls: Redesigns, Trust, Stats Errors**

+ Full timestamped outline available in the app

Show Notes

Brought to you by Mixpanel—Event analytics that everyone can trust, use, and afford | Round—The private network built by tech leaders for tech leaders | Eppo—Run reliable, impactful experiments

—

Ronny Kohavi, PhD, is a consultant, teacher, and leading expert on the art and science of A/B testing. Previously, Ronny was Vice President and Technical Fellow at Airbnb, Technical Fellow and corporate VP at Microsoft (where he led the Experimentation Platform team), and Director of Data Mining and Personalization at Amazon. He was also honored with a lifetime achievement award by the Experimentation Culture Awards in September 2020 and teaches a popular course on experimentation on Maven. In today’s podcast, we discuss:

• How to foster a culture of experimentation

• How to avoid common pitfalls and misconceptions when running experiments

• His most surprising experiment results

• The critical role of trust in running successful experiments

• When not to A/B test something

• Best practices for helping your tests run faster

• The future of experimentation

—

Enroll in Ronny’s Maven class: Accelerating Innovation with A/B Testing at https://bit.ly/ABClassLenny. Promo code “LENNYAB” will give $500 off the class for the first 10 people to use it.

—

Find the full transcript at: https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing

—

Where to find Ronny Kohavi:

• Twitter: https://twitter.com/ronnyk

• LinkedIn: https://www.linkedin.com/in/ronnyk/

• Website: http://ai.stanford.edu/~ronnyk/

—

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• Twitter: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/