Machine Learning · AI216

Linear Regression: Model, Cost Function and the Closed-Form Solution

Machine Learning Study guides

Linear regression predicts a value as a weighted sum of input features plus a bias; training finds the parameters that minimise the mean squared error.

The model is y-hat = theta0 + theta1 x1 + ... + thetan xn, where n is the number of features, xi is the i-th feature value and thetaj the j-th parameter (theta0 is the bias term). In vectorized form y-hat = h(x) = theta . x, where h is the hypothesis, the prediction function.

Training means setting the parameters so the model best fits the training set. We need to find theta that minimises the cost function. In practice it is simpler to minimise the mean squared error (MSE) than the root mean squared error (RMSE), and both give the same theta because the square root is monotonic. The MSE for linear regression is convex.

Two ways to train: a closed-form equation (normal equation) that directly computes the best parameters, or iterative optimization with gradient descent. Scikit-Learn LinearRegression uses SVD to compute the pseudoinverse, which is about O(n squared) in the number of features and is more efficient than the normal equation, handling edge cases well. Both get slow when the number of features is very large, where gradient descent is preferred.

Key points

  • y-hat = theta . x with a bias theta0.
  • Minimise MSE (equivalent to minimising RMSE).
  • Closed form (normal equation or SVD pseudoinverse) versus gradient descent.
  • SVD in LinearRegression is about O(n squared) in the number of features.
  • MSE cost is convex: one global minimum.

Worked example

Fit y = theta0 + theta1 x to points (1,2), (2,4), (3,6). The least-squares solution is theta0 = 0 and theta1 = 2, so the prediction for x = 4 is 8 and the MSE is 0.

Common questions

Why minimise MSE instead of RMSE?

The value of theta that minimises MSE also minimises RMSE, and MSE is simpler to differentiate.

Does linear regression need gradient descent?

Not always. The closed-form solution computes the parameters directly, but gradient descent scales better to a very large number of features.

Practice questions on this topic

Try these, then open the app to check your answers with explanations.

  1. If we train a Linear Regression model on the California housing data (book example), it underfits the training data.
    • A. true
    • B. false
  2. For a convex cost function such as the MSE of Linear Regression, in Gradient Descent, once the gradient is.......... you have reached a minimum.
    • A. zero
    • B. one
    • C. three
    • D. None of them
  3. To train linear regression model, use closed-form that directly computes the model parameters
    • A. true
    • B. false
  4. For the convex MSE cost of Linear Regression, GD is guaranteed to approach arbitrarily close the global minimum (if you wait long enough and the learning rate is not too high).
    • A. true
    • B. false
  5. yˆ = hθ(x) = θ.x in Linear Regression Model Prediction (Vectorized Form), hθ is refer to the............
    • A. model’s parameter vector.
    • B. instance’s feature vector.
    • C. hypothesis function.
    • D. None of them
  6. Linear Regression is a polynomial model that makes a prediction by simply computing a weighted sum of the input features
    • A. true
    • B. false