Choose the right split so your offline metrics predict real‑world performance. Learn how to match validation to data shape, leakage risks, and deployment reality.
For time‑ordered data, which validation is most appropriate to avoid look‑ahead bias?
leave‑one‑out cross‑validation
k‑fold cross‑validation with random shuffles
rolling (expanding) time‑series cross‑validation
stratified k‑fold by class only
Nested cross‑validation is primarily used to ______.
handle extreme class imbalance automatically
combine train and test into a single fold
separate hyperparameter tuning from unbiased performance estimation
speed up grid searches on large models
When classes are imbalanced, a good default is to use ______ splits to preserve prevalence in each fold.
stratified
leave‑p‑out
exhaustive
blocked
A leakage red flag is computing target‑derived statistics on the full dataset before splitting, such as ______.
dropping constant columns
setting a fixed random seed
shuffling the training fold only
normalizing with a mean computed using all rows
For products with weekly seasonality and promotions, a validation window should ______.
span full seasonal cycles and include representative promo periods
use random days to break patterns
be as short as possible to maximize sample count
exclude holidays to reduce variance
When data drifts over time, a useful scheme is ______ validation to weight recent performance more.
repeated leave‑one‑out
sliding window
stratified shuffle
blocked by user ID once
Calibrating classification thresholds on the validation set and then reporting test AUC is ______.
acceptable because AUC is threshold‑free and test set remains untouched
invalid because calibration requires the test labels
the same as training on the test set
leakage because any validation use is forbidden
For grouped observations (e.g., multiple rows per customer), a safer split is ______.
time‑series CV with random order
grouped k‑fold that keeps each group in a single fold
pure row‑wise k‑fold
leave‑group‑out with groups appearing in train and test
If you must compare offline validation to an A/B test, the most consistent pairing is ______.
use the same success metric and horizon in both evaluations
shorten the A/B horizon to speed results
change the population to reduce variance
use accuracy offline and revenue in A/B
To report a single, stable number from repeated CV runs, it’s best to share ______.
only the best fold’s score
the median training loss
the maximum AUC observed during tuning
the mean and confidence interval across folds and repeats
Starter
Keep practicing split types, leakage checks, and temporal folds.
Solid
You understand when to use CV variants and how to mirror production constraints.
Expert!
You can design leak‑free, decision‑ready validation for any dataset.
