Cross-validation estimates how well a statistical or machine-learning model will predict data it has not seen. It matters because a model can fit its training data well while performing poorly on new cases.
Cross-validation is a family of resampling and sample-splitting methods for assessing how results from a statistical analysis generalize to an independent dataset. It is especially useful when the goal is prediction, but it can also help assess a fitted model's quality and the stability of its parameters.
All flashcards
What is cross-validation?
A set of resampling and sample-splitting methods that estimates how well a model or analysis generalizes to independent data.
What is a training set?
The portion of the data used to fit a model and estimate its parameters.
What is a validation set?
Held-out data used to assess predictions from a model that was fitted without those observations.
Why is training performance often optimistic?
The model was fitted to the training observations, so it may capture their accidental features and appear to perform better there than on independent data.
What does cross-validation MSE measure?
It summarizes squared prediction errors on held-out observations across validation rounds, providing an estimate of out-of-sample error.
Quiz questions
In k-fold cross-validation, how is each round's validation set chosen?Answer: One fold is held out while the other folds are used for training.
What is the purpose of stratifying folds in binary classification?Answer: To keep the class proportions approximately similar across folds.
In nested cross-validation, what is the outer loop used for?Answer: Estimating performance of the model-selection procedure on held-out data.
Why should scaling be fitted separately within each training fold?Answer: To prevent information from held-out observations from influencing preprocessing.
Why can random cross-validation splits be misleading for spatial data?Answer: Nearby training and validation observations may be correlated.