Skip to main content
Dantes.io Logo
Dantes.io

Best resources to learnA/B Testing & Experimentation

Study sheet

A/B testing measures the causal effect of product changes. You will learn experiment design, sample size and power, statistical pitfalls, metrics and how experimentation platforms work in practice.

8 resources · · How we curate

🎓Courses(1)

course

A/B Testing (Udacity, taught by Google)

Five-lesson free course taught by Google data scientists covering metric selection and validation, ethics for experiment participants, experiment design with sample size and power, and analysis of results. Closes with a project sizing and interpreting a realistic funnel experiment.

beginner🇬🇧

📚Books(1)

bookPaid

Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

The field's standard text, by experimentation leaders at Microsoft, Google and LinkedIn. Covers Overall Evaluation Criterion design, randomization units, A/A tests, sample ratio mismatch, and platform architecture, so a reader can design and audit trustworthy experiments end to end.

intermediate🇬🇧

🌐Websites(1)

website

How Not To Run An A/B Test

Short essay showing how repeatedly checking a running test and stopping when it looks significant inflates the false positive rate from 5% to over 25%. Explains the fixed-sample assumption and the sequential and Bayesian designs that legitimately permit early stopping. The one statistical pitfall that ruins more real experiments than any other, explained in a form a beginner can absorb in ten minutes, with a correct account of why fixed-horizon significance breaks and what the valid alternatives are.

beginner🇬🇧

📄Papers(5)

scroll for more →
paper

Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology

Peer-reviewed review surveying the statistics of industrial A/B testing: variance reduction, sequential and always-valid inference, heterogeneous treatment effects, interference between units, and multiple testing. Maps which method addresses which problem and where open research questions remain. The natural next step after the book for anyone who will actually build or defend methodology.

advanced🇬🇧
paper

Overlapping Experiment Infrastructure: More, Better, Faster Experimentation

Google's KDD 2010 paper describing the layers-and-domains design that lets one search query participate in many concurrent experiments without confounding them, plus the diversion rules, tooling and review process around it. The reference account of experimentation platform architecture.

advanced🇬🇧
paper

A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments

KDD 2022 paper dismantling misconceptions that vendors and agencies actively promote: misread p-values, peeking and false always-valid claims, underpowered tests, and sample ratio mismatch. Ends with concrete guardrails platform designers should enforce so experimenters cannot make these errors. The corrective that makes the rest of the topic safe to read: it names, with statistical reasoning, exactly the intuitions that CRO blogs and A/B tool marketing teach wrongly.

advanced🇬🇧
paper

Online Controlled Experiments and A/B Tests (Encyclopedia of Machine Learning and Data Science)

Thirteen-page Springer reference entry defining controlled experiments, OEC choice, randomization units, A/A validation and common pitfalls, illustrated with a real Bing ads experiment where revenue rose while user engagement metrics degraded. A compact free route into the field's core argument.

intermediate🇬🇧
paper

A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook

Compares standard attribution methods against randomized controlled trials using 12 Facebook lift studies covering 435 million user-study observations. Observational approaches, including matching on thousands of behavioral variables, frequently failed to recover experimental results, often overstating advertising effects substantially.

advanced🇬🇧
Share this topic

More in Data Science

See all 35 topics in Data Science →

Frequently Asked Questions

What are the best free resources to learn A/B Testing & Experimentation?

Dantes has curated 8 resources for A/B Testing & Experimentation, including 1 books, 1 courses, 1 websites, 5 papers. All resources are hand-picked for quality — no algorithmic filler. Browse the full list above to find the format that works best for you.

Is A/B Testing & Experimentation hard to learn?

A/B Testing & Experimentation is approachable at the beginner level — there are resources here specifically for those starting from scratch. As you progress, intermediate and advanced material is also available to take your skills further.

What types of A/B Testing & Experimentation learning resources are available on Dantes?

For A/B Testing & Experimentation, Dantes has curated 1 books, 1 courses, 1 websites, 5 papers. Each resource type serves a different learning style: videos and YouTube for visual learners, books for depth, courses for structured progression, and websites for quick reference.

How does Dantes select A/B Testing & Experimentation resources?

Dantes is an algorithm-free learning directory. Resources are hand-curated based on quality, accuracy, and usefulness — not engagement metrics or paid placements. The goal is to surface the best learning material for A/B Testing & Experimentation, whether it's a free YouTube series, a classic textbook, or an open courseware from a top university.

Go deeper

Test your understanding of A/B Testing & Experimentation

Explain it out loud. An AI tutor listens and asks questions that expose gaps you didn't know you had.