Examine the trade‑offs between statistical power and opportunity cost when choosing test‑control proportions.
Larger holdout size increases statistical power but also raises ______ cost.
CPC
production
opportunity
impression
Foregoing ads on bigger control group sacrifices potential sales.
Meta’s 2025 guide suggests at least ______ users per cell for reliable lift tests.
10
1k
1
100k
Sample size ensures effect detection with reasonable confidence.
Power rises with holdout size because variance of control mean falls as 1/____.
n
log n
n²
sqrt(n)
Standard error decreases with square root of sample size, boosting test sensitivity.
A 10% holdout is common when baseline conversion rate exceeds ______ %.
50
5
0.01
0.1
Higher base rates need fewer withheld users to observe lift.
Small holdout fractions risk contamination from ______ effects.
pixel
font
network
cookie
Interference can spill over when test and control users interact.
Geo lift tests use ≥20 geos per cell instead of % holdouts because each geo is relatively ______.
cookie-based
identical
static
independent
Geographic separation reduces contamination enabling smaller share by headcount.
Holdout duration should balance quick reads with enough ______ accumulation.
DNS
impressions
pixels
conversion
Ending too soon leaves under‑powered results despite adequate holdout share.
Sequential testing inflates type‑I error; pre‑planning a fixed holdout window preserves ______.
lambda
gamma
alpha
beta
Stopping rules must be defined to maintain statistical validity.
Dynamic holdouts periodically swap users to control to mitigate ______ bias.
ISP
creative
seasonal
audio
Rotating holdouts distributes seasonality equally across groups.
Opportunity cost of holdouts can be estimated by applying historical ______ ROAS to withheld spend.
blended
incremental
search
cookie
This quantifies revenue forgone due to withholding ads.
Starter
Review the basics.
Solid
Nice work—refine the details.
Expert!
Exceptional command of the topic.
