Overfitting, Generalization, and Methods for Controlling Model Complexity

All flashcards

Overfitting

Fitting a dataset too closely, including its noise, in a way that may impair performance on new data.

Generalization

A model's ability to perform well on observations beyond those used to fit it.

Validation set

Data not used to fit a model, used to estimate how it may perform on unseen cases.

Bias–variance tradeoff

The need to balance errors from overly simple models against errors from models too sensitive to their training sample.

Benign overfitting

A special case in which a model fits noisy training data perfectly yet still generalizes well.

Quiz questions

1. If a validation set is repeatedly used to choose models, what can help preserve an independent final assessment?Answer: Evaluate on a separate test set that was held out from tuning.
2. How does cross-validation estimate a model's performance beyond one training sample?Answer: It fits and evaluates the model across different splits of the available data.
3. What does regularization do to discourage unnecessary model complexity?Answer: It adds a penalty for excessive complexity or extreme parameter values.
4. During neural-network training, what does dropout do?Answer: It probabilistically removes inputs or units during training.
5. What characterizes benign overfitting?Answer: A model fits noisy training data perfectly but can still generalize well under particular conditions.