Confidence intervals turn a single sample estimate into a range that likely captures the true value. Mastering them prevents over‑confidence in noisy A/B test data..
What does a 95% confidence interval for a sample mean claim?
The sample mean equals the true mean 95% of the time
We are 95% confident the interval contains the true mean
There is a 95% chance the sample mean is correct
95% of individual data points fall in the interval
Narrower confidence intervals generally result from ______.
lower confidence levels
larger sample sizes
shorter experiment duration
higher variance data
If a 90% CI for lift excludes zero, the result is ______ at the 10% level.
practically insignificant
biased upward
statistically significant
over‑fit
Increasing the confidence level from 90% to 99% makes the interval ______.
remain unchanged
wider
shift upward
narrower
For proportions, Wilson intervals outperform Wald intervals because they ______.
assume normality of raw counts
require Bayesian priors
ignore variance
maintain coverage even near 0 or 1
Bootstrapped confidence intervals rely on ______ sampling of the observed data.
Latin hypercube
importance
systematic without replacement
resampling with replacement
A confidence interval that overlaps the business minimum detectable effect may still be useful because it ______.
removes all variance
confirms the null
guarantees ROI
quantifies uncertainty around the lift estimate
Which statistic is at the center of a two‑sided CI for an A/B difference in means?
the pooled standard deviation
the median difference
the observed difference between variant means
zero
When sample size is small and variance unknown, confidence intervals use the ______ distribution.
t‑
chi‑square
normal
F
The term ‘coverage probability’ for a CI describes ______.
the chance any interval overlaps others
how often intervals from repeated samples contain the true value
percent of data within one SD
page‑view reach
Starter
Solidify the basics of the topic before running live tests.
Solid
You understand the core ideas—focus on edge‑cases to boost accuracy.
Expert!
Outstanding—you can teach your team and steer experimentation strategy.
