Ordinary Least Squares

/ˈɔːr.dən.er.i liːst skwɛərz/

noun … “fitting a line to tame the scatter.”

Ordinary Least Squares (OLS) is a fundamental method in statistics and regression analysis used to estimate the parameters of a linear model by minimizing the sum of squared differences between observed outcomes and predicted values. It provides the best linear unbiased estimates under classical assumptions, allowing analysts to quantify relationships between predictor variables and a response variable while assessing the strength and direction of these relationships.

Formally, for a linear model Y = Xβ + ε, where Y is the vector of observations, X is the matrix of predictors, β is the vector of coefficients, and ε is the error term, OLS estimates β̂ by minimizing Σ (Yᵢ - Xᵢβ)². The solution is given by β̂ = (XᵀX)⁻¹XᵀY when XᵀX is invertible. The method assumes linearity, independence of errors, homoscedasticity (constant Variance of errors), and normality of residuals for inference purposes.

Ordinary Least Squares underpins many statistical and machine learning applications. It is the core of Linear Regression, used for prediction, feature evaluation, and hypothesis testing. OLS estimates interact with concepts like Variance, covariance matrices (Covariance Matrix), and expectation values (Expectation Value) to assess uncertainty, confidence intervals, and significance of coefficients. It is also a building block for generalized linear models, ridge regression, and principal component regression.

Example conceptual workflow for OLS regression:

collect dataset with response and predictor variables
verify assumptions: linearity, independence, constant variance
construct predictor matrix X and response vector Y
compute OLS estimator: β̂ = (XᵀX)⁻¹XᵀY
analyze residuals to check model fit and assumptions
use fitted model for prediction or inference

Intuitively, Ordinary Least Squares is like stretching a tightrope through a scatter of points: the line seeks the path that stays as close as possible to all points simultaneously. Each squared deviation acts as a tension force, guiding the line toward balance, producing a stable and interpretable summary of how predictors influence outcomes.