--- title: A/B Testing & Experimentation description: A/B testing measures the causal effect of product changes. You will learn experiment design, sample size and power, statistical pitfalls, metrics and how experimentation platforms work in practice. category: programming-tech subcategory: data-science difficulty: beginner, intermediate, advanced url: /subject/ab-testing-and-experimentation --- # A/B Testing & Experimentation A/B testing measures the causal effect of product changes. You will learn experiment design, sample size and power, statistical pitfalls, metrics and how experimentation platforms work in practice. ## Available Resources 1 Books • 1 Courses • 1 Websites • 5 Papers ## Papers ### 1. Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology **Author:** Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi, Nathaniel Stevens Peer-reviewed review surveying the statistics of industrial A/B testing: variance reduction, sequential and always-valid inference, heterogeneous treatment effects, interference between units, and multiple testing. Maps which method addresses which problem and where open research questions remain. The natural next step after the book for anyone who will actually build or defend methodology. **Difficulty:** Advanced | **Language:** English | **Price:** Free **Link:** https://arxiv.org/abs/2212.11366 **Tags:** ab-testing, online-experiments, variance-reduction, sequential-testing, causal-inference, statistics ### 2. Overlapping Experiment Infrastructure: More, Better, Faster Experimentation **Author:** Diane Tang, Ashish Agarwal, Deirdre O'Brien, Mike Meyer Google's KDD 2010 paper describing the layers-and-domains design that lets one search query participate in many concurrent experiments without confounding them, plus the diversion rules, tooling and review process around it. The reference account of experimentation platform architecture. **Difficulty:** Advanced | **Language:** English | **Price:** Free **Link:** https://research.google/pubs/overlapping-experiment-infrastructure-more-better-faster-experimentation/ **Tags:** ab-testing, experimentation-platforms, experiment-design, overlapping-experiments, data-infrastructure ### 3. A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments **Author:** Ron Kohavi, Alex Deng, Lukas Vermeer KDD 2022 paper dismantling misconceptions that vendors and agencies actively promote: misread p-values, peeking and false always-valid claims, underpowered tests, and sample ratio mismatch. Ends with concrete guardrails platform designers should enforce so experimenters cannot make these errors. The corrective that makes the rest of the topic safe to read: it names, with statistical reasoning, exactly the intuitions that CRO blogs and A/B tool marketing teach wrongly. **Difficulty:** Advanced | **Language:** English | **Price:** Free **Link:** https://exp-platform.com/abtestingintuitionbusters/ **Tags:** ab-testing, p-values, statistical-power, sample-ratio-mismatch, experiment-pitfalls ### 4. Online Controlled Experiments and A/B Tests (Encyclopedia of Machine Learning and Data Science) **Author:** Ron Kohavi, Roger Longbotham Thirteen-page Springer reference entry defining controlled experiments, OEC choice, randomization units, A/A validation and common pitfalls, illustrated with a real Bing ads experiment where revenue rose while user engagement metrics degraded. A compact free route into the field's core argument. **Difficulty:** Intermediate | **Language:** English | **Price:** Free **Link:** https://exp-platform.com/Documents/2023-03-11EncyclopeiaMLDSABTestingFinal.pdf **Tags:** ab-testing, controlled-experiments, overall-evaluation-criterion, randomization, experiment-design ### 5. A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook **Author:** Brett Gordon, Florian Zettelmeyer, Neha Bhargava, Dan Chapsky Compares standard attribution methods against randomized controlled trials using 12 Facebook lift studies covering 435 million user-study observations. Observational approaches, including matching on thousands of behavioral variables, frequently failed to recover experimental results, often overstating advertising effects substantially. **Difficulty:** Advanced | **Language:** English | **Price:** Free **Link:** https://www.kellogg.northwestern.edu/faculty/gordon_b/files/kellogg_fb_whitepaper.pdf **Tags:** advertising-measurement, attribution, randomized-experiments, causal-inference, facebook-ads ## Websites ### 1. How Not To Run An A/B Test **Author:** Evan Miller Short essay showing how repeatedly checking a running test and stopping when it looks significant inflates the false positive rate from 5% to over 25%. Explains the fixed-sample assumption and the sequential and Bayesian designs that legitimately permit early stopping. The one statistical pitfall that ruins more real experiments than any other, explained in a form a beginner can absorb in ten minutes, with a correct account of why fixed-horizon significance breaks and what the valid alternatives are. **Difficulty:** Beginner | **Language:** English | **Price:** Free **Link:** https://www.evanmiller.org/how-not-to-run-an-ab-test.html **Tags:** ab-testing, sequential-testing, false-positives, statistical-significance, experiment-pitfalls ## Books ### 1. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing **Author:** Ron Kohavi, Diane Tang, Ya Xu The field's standard text, by experimentation leaders at Microsoft, Google and LinkedIn. Covers Overall Evaluation Criterion design, randomization units, A/A tests, sample ratio mismatch, and platform architecture, so a reader can design and audit trustworthy experiments end to end. **Difficulty:** Intermediate | **Language:** English | **Price:** Paid **Link:** https://www.amazon.com/dp/1108724264?tag=edmonddante07-20 **Tags:** ab-testing, controlled-experiments, experiment-design, experimentation-platforms, product-analytics ## Courses ### 1. A/B Testing (Udacity, taught by Google) **Author:** Carrie Grimes, Caroline Buckey, Diane Tang Five-lesson free course taught by Google data scientists covering metric selection and validation, ethics for experiment participants, experiment design with sample size and power, and analysis of results. Closes with a project sizing and interpreting a realistic funnel experiment. **Difficulty:** Beginner | **Language:** English | **Price:** Free **Link:** https://www.udacity.com/course/ab-testing--ud257 **Tags:** ab-testing, experiment-design, sample-size, metrics, statistical-power --- *This content is part of Dantes.io - Your Treasure Map to Knowledge* *Curated by humans at Dantes.io. Personal study use welcome; republishing this curation requires permission (team@dantes.io).* View this page online: https://dantes.io/subject/ab-testing-and-experimentation