Machine-Learning Regularization: Penalties, Sparsity, and Shared Structure

All flashcards

What is regularization?

A process that favors simpler, more stable, or otherwise restricted solutions. It is used to stabilize ill-posed problems and to reduce overfitting so that models can generalize better.

What is the difference between explicit and implicit regularization?

Explicit regularization adds a penalty, prior term, or constraint directly to the optimization problem. Implicit regularization comes from choices such as the training procedure, stopping time, loss, or model construction.

What does λ control in a regularized objective?

λ controls the relative importance of the regularization penalty compared with fitting the observed data. Increasing it generally places greater emphasis on the penalty.

How does regularization connect to Bayesian statistics?

The data-fit term can represent a likelihood and the regularizer can represent a prior. Their combination yields a posterior that incorporates both observations and prior preferences.

What is generalization, and why is it important?

Generalization is a model’s ability to make accurate predictions on data not used for training. It matters because low training error alone does not ensure useful performance on new examples.

Quiz questions

1. What is overfitting?Answer: When a model fits details or noise in its training data but performs poorly on new data.
2. What is the main purpose of regularization in machine learning?Answer: To favor solutions that generalize well rather than merely fitting training examples.
3. Which choice is an example of explicit regularization?Answer: Adding a coefficient penalty directly to the optimization objective.
4. Which choice is an example of implicit regularization?Answer: Early stopping, which limits training without adding a penalty term to the objective.
5. In a Bayesian interpretation, what does a regularization term often represent?Answer: A prior preference over model parameters or functions, combined with the data likelihood.