Test your grasp of MAPE, hold‑out accuracy, R‑squared caveats, and Bayesian diagnostics for model validation.
Mean Absolute Percentage Error (MAPE) compares model predictions to ______.
actual observed values
Bayes prior
moving average
CTR
MAPE expresses average error as a percent of real sales, offering scale‑free interpretability.
A hold‑out period evaluates the model’s ability to ______ future data.
backfit
ad‑serve
forecast
normalise
Holding out final weeks tests generalisation rather than overfitting the estimation window.
R‑squared can be misleadingly high in MMM if ______ variables dominate variation.
cookie
trend
creative
week
Long‑term trend inflates R² even when media coefficients are poor; thus multiple metrics are used.
2025 Facebook Robyn 2.4 adds Deviation Penalty which penalises forecasts that exceed ______ deviation.
10 %
50 %
0 %
1 %
Penalty discourages models that occasionally miss by large margins while achieving good average fit.
Normalized Root Mean Squared Error (NRMSE) divides RMSE by ______ to enable comparison.
range of actuals
sample size
CPC
mean spend
NRMSE expresses error relative to data scale, aiding multi‑product roll‑ups.
Cross‑validation in MMM often uses **time‑series split** to respect ______ order.
random
chronological
alph numeric
device
Random shuffles break temporal dependency; sequential folds keep causality intact.
Model residuals should resemble white noise; detecting autocorrelation may require adding ______.
hashtags
pixel load
lagged variables
extra creative
Lagged predictors absorb unexplained serial dependence, improving fit.
Cook’s distance identifies influential observations; high values may correspond to ______ outliers.
device
promo
cookie
ctr
Sales spikes from promotions can skew coefficient estimates; analysts down‑weight or model separately.
Out‑of‑sample MAPE below ______ is often targeted for weekly MMM forecasts.
25 %
10 %
1 %
50 %
Single‑digit MAPE indicates robust predictive accuracy suitable for decision support.
Pareto Smoothed Importance Sampling (PSIS) k‑hat statistic above 0.7 in Bayesian MMM suggests ______.
good fit
problematic influence of observations
negative ROI
perfect independence
High k‑hat values warn that few points dominate likelihood, undermining reliable inference.
Starter
Review the basics.
Solid
Nice work—refine the details.
Expert!
Exceptional command of the topic.
