How not to run an A/B test
Listed inA/B TestingMeasure & Iterateon
Evan Miller on peeking at results and stopping early — the mistake that produces confident, wrong conclusions.
External
Evan Miller · evanmiller.org#experimentation
#statistics
What happened after launch: instrumenting the experience, running experiments, and turning the results into the next round of work.
11 articles
Listed inA/B TestingMeasure & Iterateon
Evan Miller on peeking at results and stopping early — the mistake that produces confident, wrong conclusions.
Listed inFunnel AnalysisMeasure & Iterateon
Locating the step that loses people, and resisting the story you attach to the drop.
Listed inExperimentationMeasure & Iterateon
Running a program rather than one-off tests: hypotheses, guardrails, and a decision rule.
Listed inHEART FrameworkMeasure & Iterateon
Happiness, engagement, adoption, retention, task success — with the goals-signals-metrics step.
Listed inMultivariate TestingMeasure & Iterateon
Testing several changes at once, the interaction effects, and the traffic it demands.
Listed inInstrumentation PlansMeasure & Iterateon
Deciding what to track before the build, so the data exists when the question arrives.
Listed inA/B TestingMeasure & Iterateon
Randomisation, exposure, and significance — plus the traps that produce confident wrong answers.
Listed inContinuous ImprovementMeasure & Iterateon
Feeding measurement back into discovery so the loop closes instead of restarting.
Listed inProduct AnalyticsMeasure & Iterateon
Events, properties, and the questions a well-instrumented product can actually answer.
Listed inCohort & Retention AnalysisMeasure & Iterateon
Reading retention curves and cohort behaviour as the clearest signal design work has changed anything.
Listed inFeature FlagsMeasure & Iterateon
Staged rollout, holdbacks, and kill switches as design tools rather than deployment plumbing.