Overlapping Experiment Infrastructure: More, Better, Faster Experimentation
by Diane Tang, Ashish Agarwal, Deirdre O'Brien, Mike Meyer · Google Research
Google's KDD 2010 paper describing the layers-and-domains design that lets one search query participate in many concurrent experiments without confounding them, plus the diversion rules, tooling and review process around it. The reference account of experimentation platform architecture.
More resources on A/B Testing & Experimentation
Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology
Peer-reviewed review surveying the statistics of industrial A/B testing: variance reduction, sequential and always-valid inference, heterogeneous treatment effects, interference between units, and multiple testing. Maps which method addresses which problem and where open research questions remain. The natural next step after the book for anyone who will actually build or defend methodology.
A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments
KDD 2022 paper dismantling misconceptions that vendors and agencies actively promote: misread p-values, peeking and false always-valid claims, underpowered tests, and sample ratio mismatch. Ends with concrete guardrails platform designers should enforce so experimenters cannot make these errors. The corrective that makes the rest of the topic safe to read: it names, with statistical reasoning, exactly the intuitions that CRO blogs and A/B tool marketing teach wrongly.
Online Controlled Experiments and A/B Tests (Encyclopedia of Machine Learning and Data Science)
Thirteen-page Springer reference entry defining controlled experiments, OEC choice, randomization units, A/A validation and common pitfalls, illustrated with a real Bing ads experiment where revenue rose while user engagement metrics degraded. A compact free route into the field's core argument.
How Not To Run An A/B Test
Short essay showing how repeatedly checking a running test and stopping when it looks significant inflates the false positive rate from 5% to over 25%. Explains the fixed-sample assumption and the sequential and Bayesian designs that legitimately permit early stopping. The one statistical pitfall that ruins more real experiments than any other, explained in a form a beginner can absorb in ten minutes, with a correct account of why fixed-horizon significance breaks and what the valid alternatives are.
Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
The field's standard text, by experimentation leaders at Microsoft, Google and LinkedIn. Covers Overall Evaluation Criterion design, randomization units, A/A tests, sample ratio mismatch, and platform architecture, so a reader can design and audit trustworthy experiments end to end.
A/B Testing (Udacity, taught by Google)
Five-lesson free course taught by Google data scientists covering metric selection and validation, ethics for experiment participants, experiment design with sample size and power, and analysis of results. Closes with a project sizing and interpreting a realistic funnel experiment.