Regularization and Ridge Regression
Ridge regression is linear regression with an l2 regularization term added to the cost function, which keeps the weights small and reduces overfitting.
A good way to reduce overfitting is to constrain the model: the fewer degrees of freedom it has, the harder it is to overfit. For a linear model this means constraining the weights.
Ridge regression adds a regularization term equal to alpha times the sum of the squared weights (half of the l2 norm squared) to the cost function during training only. This forces the learning algorithm to keep the weights as small as possible while still fitting the data. The hyperparameter alpha controls how much to regularize: alpha = 0 gives plain linear regression, and a very large alpha makes all weights end up very close to zero, giving a flat line through the mean of the data. The bias term theta0 is not regularized.
The regularization term is added only for training. Once trained, you evaluate the model with the unregularized performance measure. Scale the data first (for example StandardScaler) because ridge is sensitive to the scale of the input features.
Key points
- Ridge = MSE + alpha x sum of squared weights.
- Alpha is the regularization hyperparameter; larger alpha means simpler model.
- Regularization is applied during training only.
- Scale features before ridge.
- The bias term is not regularized.
Worked example
With alpha = 0 a degree-10 polynomial regression overfits wildly. Increase alpha and the curve flattens and stops chasing noise; with alpha too large it becomes a flat line (underfit).
Common questions
What is the purpose of the regularization term in ridge regression?
To force the weights to stay small, which constrains the model and reduces overfitting.
What happens if alpha is 0?
Ridge regression becomes ordinary linear regression.
Practice questions on this topic
Try these, then open the app to check your answers with explanations.
- What is purpose of regularization term in Ridge Regression?
- A. To ignore the impact of model weights during training
- B. To penalize large model weights by adding α∑θ^2 to the cost function.
- C. To increase the size of model weights
- D. All of them are false
Related topics
- Gradient Descent: Batch, Stochastic and Mini-batch
- Linear Regression: Model, Cost Function and the Closed-Form Solution
- What is Machine Learning?
- Supervised vs Unsupervised vs Semi-supervised vs Reinforcement Learning
- Overfitting and Underfitting in Machine Learning
- Batch Learning vs Online Learning
- All topics: Machine Learning
BASMAGLY is an independent student project, not an official Delta University site. Check lectures and your instructor for the final word.