Test your knowledge of common a/b testing pitfalls.
Stopping a test the moment the p‑value dips below 0.05 most often inflates the risk of ______.
false positives
power
false negatives
sample ratio mismatch
Peeking adds repeated looks at the data, increasing the cumulative chance of a Type I error. Modern platforms discourage early stopping unless you use alpha‑spending methods.
A sample‑ratio mismatch (SRM) usually signals an underlying ______ issue.
seasonality
Bayesian prior
implementation / tracking
statistical modelling
SRM means the traffic split delivered differs from the plan, often due to mis‑routing or analytics filters. Trusting results without fixing SRM can completely invalidate the experiment.
Testing too many variations at once without correction mainly raises the danger of the ______ problem.
segmentation creep
multiple comparisons
carry‑over bias
novelty effect
Each additional comparison adds another chance to see a spurious winner. Adjustments like Holm‑Bonferroni or false discovery rate control keep overall error in check.
Running a test across only one week of a highly seasonal business risks ______ bias.
geo
temporal / seasonality
device
instrumentation
Short windows may capture payday spikes or weekend lulls that do not generalise. Extending tests across full business cycles stabilises estimates.
Making many post‑hoc cuts of the data to “find” a win is an example of ______.
lift clipping
bandit exploration
Bayesian updating
p‑hacking
Searching until something appears significant defeats the premise of a pre‑registered hypothesis. It inflates Type I error and undermines reproducibility.
Failure to hold recreatable control cookies for returning visitors breaks the assumption of ______ observations.
homoscedastic
independent
normal
uniform
Users switching buckets leak treatment effects and bias variance estimates. Sticky bucketing or user‑ID assignment preserves independence.
Comparing conversion rate lifts without examining incremental revenue risks ignoring ______ significance.
practical / business
sequential
statistical
directional
A tiny lift can be statistically significant yet worth pennies, while a large monetary gain may be noisy. Decision frameworks weigh both statistics and economics.
Letting the test platform auto‑re‑allocate traffic mid‑experiment can introduce ______ bias if not accounted for.
observer
allocation
recall
reporting
Adaptive traffic changes the exposure probability over time, so naive analyses that assume fixed randomisation mis‑estimate variance.
Testing only mobile traffic when desktop drives half the revenue invites a ______ mismatch.
cookie
population
cache
metric
Results generalise only to the sampled population. Un‑represented segments may react differently, so stratified or inclusive tests are safer.
Failing to account for bot or internal traffic in metrics can create illusionary lifts due to ______ noise.
gaussian
regression‑to‑mean
sampling
non‑human
Bots inflate page views or conversions unevenly across buckets, corrupting KPI accuracy. Filtering known IPs and bot signatures preserves data quality.
Starter
Brush up on the basics and avoid costly mistakes.
Solid
You know your stuff—refine a few nuances for mastery.
Expert!
Outstanding! You could teach this topic.
