Machine Learning · AI216

Overfitting and Underfitting in Machine Learning

Machine Learning Study guides

Overfitting means a model fits the training data very well but does not generalize to new data; underfitting means the model is too simple to learn the structure of the data.

Overfitting happens when the model is too complex for the amount and noise of the training data. It memorises noise and outliers, so training error is low and error on new data is high. To fight it: simplify the model (fewer parameters, fewer features), get more training data, reduce noise by fixing errors and removing outliers, or constrain the model with regularization.

Underfitting is the opposite. The model is too simple to capture the underlying pattern, so it does badly even on the training data. To fix it: select a more powerful model with more parameters, feed better features to the learning algorithm (feature engineering), or reduce the constraints, for example lower the regularization hyperparameter.

Regularization means constraining a model to make it simpler and reduce the risk of overfitting. The amount of regularization is controlled by a hyperparameter set before training.

Key points

  • Overfitting: good on training, bad on new data.
  • Underfitting: bad on both.
  • Overfitting fixes: simpler model, more data, less noise, more regularization.
  • Underfitting fixes: stronger model, better features, less regularization.
  • A hyperparameter belongs to the learning algorithm, not the model, and is set before training.

Worked example

Fitting a degree-15 polynomial to 12 points passes through every point (overfit) but swings wildly between them. Fitting a straight line to data that clearly curves misses the shape (underfit).

Common questions

Does more regularization reduce overfitting?

Yes, more regularization constrains the model more and lowers overfitting. Too much causes underfitting.

Is low training error always good?

No. Low training error with high error on new data signals overfitting.

Practice questions on this topic

Try these, then open the app to check your answers with explanations.

  1. It occurs when your model is too simple to learn the underlying structure of the data.
    • A. Overfitting the Training Data
    • B. fitting the Training Data
    • C. Underfitting the training data
    • D. regularization
  2. What are the Solutions for Overfitting?
    • A. Simplify the mode
    • B. Gather more training data
    • C. Reduce the noise in the training data
    • D. All of them are true
  3. Underfitting the Training Data It occurs when your model is too simple to learn the underlying structure of the data.
    • A. true
    • B. false
  4. One of the Solutions for underfitting reduce the regularization hyperparameter.
    • A. true
    • B. false
  5. One of the solution of overfitting Simplify the model by selecting one with ............. Parameters.
    • A. fewer
    • B. higher
    • C. any
    • D. good
  6. Constraining a model to make it simpler and reduce the risk of overfitting is called .................
    • A. regularization
    • B. Overfitting the Training Data
    • C. Underfitting the training data
    • D. feature reduction