Great experiments start with the right sample and enough statistical power. Sharpen your intuition on detecting effects without wasting traffic.
Statistical power is defined as 1 minus ______.
p‑value
Type II error (beta)
effect size
alpha
For a fixed effect size, increasing sample size will ______ power.
reverse
decrease
increase
have no effect
Minimal Detectable Effect (MDE) shrinks when you ______.
split traffic unevenly
enlarge the sample
lower power target
raise alpha
Which metric most strongly drives sample size needs in proportion tests?
page load time
number of variants
baseline conversion rate
cookie expiration
Underpowered tests mainly risk producing ______.
multiple comparisons
inflated p‑values
overfitting
false negatives
Traffic allocation that is 90/10 rather than 50/50 usually ______ required test duration.
shortens
eliminates
keeps constant
extends
A one‑tailed test generally needs ______ observations than a two‑tailed test for the same power.
double
more
identical
fewer
Variance reduction techniques like CUPED mostly allow you to ______.
reduce sample size
lower power target
inflate alpha
switch to nonparametric stats
Beta spending in sequential designs keeps overall power while ______ sample size.
guaranteeing
ignoring
potentially lowering
doubling
The primary benefit of stratified sampling is ______.
higher alpha
easier randomisation
avoiding control traffic
variance reduction within groups
Starter
Dive deeper into the fundamentals to boost your confidence.
Solid
You’ve got the core ideas—polish a few nuances for mastery.
Expert!
Outstanding—your stats savvy rivals that of a data scientist.
