Author: Ernad Mujakic
Date: 2025-07-21

The least squares method is an optimization technique that attempts to find a linear function which minimizes the sum of squared distances between observed and predicted values. This method is widely used in Regression analysis for finding the best-fit function between a matrix of features and one or more independent variables.

In a Simple Linear Regression, the model is defined as:

Where:

The objective is to minimize the sum of squared Euclidean distances:

Where:

This is equivalent to min

The most common form of the least squares method, commonly used in Linear Regression models. It simply minimizes the sum of squared residuals without any weights.

Extends ordinary least squares to handle heteroscedasticity by assigning weights to data objects based on Covariance Matrix of the residuals.

A specific type of generalized least squares, used when the dataset exhibits heteroscedasticity. WLS is more robust because it weighs the influence of each data point based on its variance.

Also known as L2 regularization, ridge regression is a type of linear regression that introduces a penalty term to the OLS cost function. Ridge regression adds the squared coefficients as a penalty to the cost function, thereby, punishing large coefficient values. This helps prevent overfitting and addresses any Multicollinearity in the independent variables.
The cost function for ridge regression defined as:

Where:

By adjusting the ridge parameter, λ, ridge regression allows for control over the bias-variance tradeoff. As increases, the bias of the model increases and the variance decreases. Optimizing this parameter can achieve the ideal balance between overfitting and underfitting model.

Also known as L1 regularization, lasso regression is a type of linear regression which, like ridge regression, introduces a penalty term to the OLS cost function. Lasso regression adds the absolute coefficient values to the cost function, this allows the coefficients of irrelevant variables to be shrunk down to 0, thereby, simplifying and regularizing the model.
The cost function for lasso regression defined as:

Where:

Unlike ridge regression, which shrinks coefficients but typically keeps all predictors in the model, lasso regression can set the coefficients of irrelevant variables exactly to zero. This makes it an effective technique for Feature Selection, that is, identifying and retaining only the most important variables.