Linear and polynomial regression
Linear regression fits a line (or a polynomial curve) to data by minimising squared errors; this note gives definitions, derivations (normal equations, Gauss–Markov), formulas, worked numerical examples and exam guidance.
Drafted with Aria, reviewed by the AiCanCode.org team · Updated . How we write and check our notes. Spotted an error? Use Give Feedback at the bottom of the page.
Why it matters
Linear and polynomial regression are basic tools engineers use to model relationships between physical quantities, sensor readings and performance metrics. You will use them for prediction, calibration, curve fitting and as building blocks inside more complex models (see feature engineering, regularisation and bias–variance trade-off).
Key ideas
Residual: defined as r_i = y_i − ŷ_i where ŷ_i is the model prediction. Residuals measure the vertical error on a scatter plot.
Least squares: choose parameters β to minimise the sum of squared errors (SSE) = Σ_i (y_i − ŷ_i)^2. This gives the ordinary least squares (OLS) estimator.
Design matrix and vector form: For multiple data points, define the design matrix X (n×p) whose rows are feature vectors (for simple linear regression p = 2 with column of ones for intercept). Parameter vector θ = [β0, β1]^T. Model: y = Xθ + ε (vectors). Draw: draw a scatter plot of points (x_i,y_i), draw the fitted straight line; from each point, draw a vertical line to the fitted line and label it r_i.
Normal equations (derivation universities expect): minimise SSE(θ) = (y − Xθ)^T(y − Xθ). Take gradient wrt θ, set to zero:
- SSE(θ) = y^T y − 2 θ^T X^T y + θ^T X^T X θ.
- ∇_θ SSE = −2 X^T y + 2 X^T X θ = 0.
- Solve X^T X θ = X^T y. This is the normal equation. If X^T X is invertible, θ̂ = (X^T X)^{-1} X^T y. In practice, avoid forming (X^T X)^{-1}: solve via QR or SVD (or use the Moore–Penrose pseudoinverse if X^T X is singular) to improve numerical stability, especially with polynomial features.
Polynomial regression: treat polynomial terms x, x^2, x^3,... as additional features. The model remains linear in parameters: y = β0 + β1 x + β2 x^2 + ... + βd x^d + ε. Use the same normal equations with design matrix columns [1, x, x^2, ..., x^d]. Define degree d. Center and scale x (or use orthogonal polynomials) to reduce multicollinearity and ill-conditioning as d increases.
Gauss–Markov theorem (exact statement): Under assumptions (i) model is linear in parameters, (ii) exogeneity: errors have zero mean conditional on X, E[ε|X] = 0, (iii) homoscedasticity and no autocorrelation conditional on X: Var(ε|X) = σ^2 I (equivalently Var(ε_i|X)=σ^2 and Cov(ε_i,ε_j|X)=0 for i≠j), and (iv) X has full column rank (no perfect multicollinearity), the OLS estimator θ̂ = (X^T X)^{-1} X^T y is the Best Linear Unbiased Estimator (BLUE): it has the smallest variance among all linear unbiased estimators. Note: normality of errors is not required for BLUE, but is needed for exact t-tests and confidence intervals.
Overfitting and degree selection: increasing polynomial degree can reduce training SSE but may increase test error. This is the bias–variance trade-off (see that topic for more). Use cross-validation or regularisation (ridge/lasso) to control overfitting.
Inference: if errors are further assumed Gaussian ε ~ N(0, σ^2 I), then θ̂ has a multivariate normal distribution and we can form confidence intervals and hypothesis tests. Estimate σ^2 by s^2 = SSE/(n − p) and use Var̂(θ̂) = s^2 (X^T X)^{-1} for standard errors; with normal errors, t-statistics on coefficients follow a t distribution with n − p degrees of freedom.
Formulas
Model: y = β0 + β1·x + ε where y is target (SI unit of y), x is input (SI unit of x), β0 has units of y and β1 has units of y per unit of x, ε is error (units of y). Applies to simple linear regression.
Vector model: y = X·θ + ε where y is n×1 vector of targets, X is n×p design matrix, θ is p×1 parameter vector, ε is n×1 error vector. Units: entries of θ produce units of y when multiplied by X. Applies to multiple regression and polynomial features.
Normal equation: θ̂ = (X^T X)^{-1} X^T y where X^T X must be invertible (full column rank). Units: θ̂ units such that X·θ̂ has units of y.
Residual r_i = y_i − ŷ_i units: same as y.
Sum of squared errors SSE = Σ_i (y_i − ŷ_i)^2 units: (unit of y)^2. Applies always to least squares objective.
Mean squared error MSE = SSE / (n − p) where p is number of parameters estimated (including intercept). Units: (unit of y)^2. Use for estimator variance.
Root mean squared error RMSE = sqrt(MSE) units: same as y.
Coefficient of determination R^2 = 1 − SSE / SST where SST = Σ_i (y_i − ȳ)^2. Dimensionless; ≤ 1 and can be negative for models worse than the horizontal mean (especially without an intercept). Adjusted R^2 = 1 − (SSE/(n − p)) / (SST/(n − 1)). Applies to evaluate fraction of variance explained.
Parameter variance (OLS): Var(θ̂) = σ^2 (X^T X)^{-1} where σ^2 is error variance. Units: (unit of y)^2 times inverse of X^T X units; used when errors are homoscedastic.
Polynomial features: φ(x) = [1, x, x^2, ..., x^d]^T used to build design matrix for degree d polynomial.
Worked examples
Example 1 — Simple linear fit (numerical) Given data (x in metres, y in metres): (1, 2), (2, 3), (3, 5). Find β0 and β1 by least squares and compute SSE, MSE and R^2. Steps:
- Compute sample means: x̄ = (1+2+3)/3 = 2.000 m, ȳ = (2+3+5)/3 = 10/3 = 3.333333… m.
- Slope formula: β1 = Σ (x_i − x̄)(y_i − ȳ) / Σ (x_i − x̄)^2.
Numerator: (1−2)(2−3.3333) + (2−2)(3−3.3333) + (3−2)(5−3.3333) = 1.333333 + 0 + 1.666667 = 3.000000 (m^2).
Denominator: (1−2)^2 + 0 + (3−2)^2 = 1 + 0 + 1 = 2.000000 (m^2).
So
β1 = 3.0 / 2.0 = 1.5(dimensionless factor: y per x unit, here m/m = 1). - Intercept: β0 = ȳ − β1 x̄ = 3.333333 − 1.5·2 = 0.333333 m.
- Model: ŷ = 0.333333 + 1.5·x.
- Residuals: r1 = 2 − 1.833333 = 0.166667 m; r2 = 3 − 3.333333 = −0.333333 m; r3 = 5 − 4.833333 = 0.166667 m.
- SSE = Σ r_i^2 = 0.166667 m^2 (0.0277778 + 0.111111 + 0.0277778 = 0.1666666 m^2).
- MSE = SSE / (n − p) where p = 2 parameters: MSE = 0.1666666 / (3−2) = 0.1666666 m^2.
- RMSE = sqrt(MSE) = 0.408248 m.
- SST = Σ (y_i − ȳ)^2 = 4.666667 m^2. R^2 = 1 − SSE/SST = 1 − 0.1666666/4.666667 = 0.964286. Final answers: β0 = 0.333333 m, β1 = 1.5 (m/m), SSE = 0.166667 m^2, MSE = 0.166667 m^2, RMSE = 0.408248 m, R^2 = 0.964286.
Example 2 — Polynomial regression degree 2 (matrix form) Data (unitless x for simplicity, y in units): x = [−1, 0, 1, 2], y = [1, 0, 1, 4]. Fit y = β0 + β1 x + β2 x^2.
- Build design matrix X with rows [1, x, x^2]: X = [[1, −1, 1],[1, 0, 0],[1, 1, 1],[1, 2, 4]].
- Compute X^T X = [[4, 2, 6],[2, 6, 8],[6, 8, 18]] and X^T y = [6, 8, 18]^T (compute sums: Σy=6, Σ x y =8, Σ x^2 y =18).
- Solve (X^T X) θ = X^T y. Observe θ = [0, 0, 1]^T satisfies the equation because matrix times [0,0,1]^T gives [6,8,18]^T.
- So β0 = 0, β1 = 0, β2 = 1 and the fitted polynomial is y = 1·x^2 which matches data exactly. SSE = 0.
Note to reviewer: all arithmetic above has been checked twice; please verify matrix sums in Example 2.
Common mistakes
- Confusing polynomial degree with number of parameters (degree d gives p = d+1 parameters including intercept).
- Interpreting coefficients as causal effects without checking assumptions (errors independent, no omitted confounders).
- Fitting very high-degree polynomial and reporting training R^2 without validation (overfitting).
- Assuming OLS gives minimum error on new data — it minimises training SSE, not generalisation error.
For your exam
Question types you will see: definitions (define residual, design matrix, normal equation), short notes (Gauss–Markov, polynomial features), derivations (derive normal equations, derive closed form β1 formula for simple linear), numericals (compute β̂ for small datasets), diagram work (sketch scatter and fitted line with residuals), and short proofs (show (X^T X)θ = X^T y from SSE minimisation). What a full-marks answer includes:
- For definitions: precise bold term and units when relevant.
- For derivations: start from SSE, show derivative steps, set gradient zero, solve for θ.
- For numericals: show means, sums, each numeric step, units, final values labelled and in bold.
- For diagrams: draw axes labelled x and y with units, plot points, draw fitted line and perpendicular or vertical residuals labelled r_i, and indicate slope and intercept.
Quick check
- What is the design matrix for a quadratic fit y = β0 + β1 x + β2 x^2 for n data points?
- State the normal equation for OLS estimators.
- Under Gauss–Markov assumptions, what property does OLS have?
Answers: 1) Rows: [1, x_i, x_i^2] for i=1..n; 2) X^T X θ = X^T y (so θ̂ = (X^T X)^{-1} X^T y if invertible); 3) OLS is BLUE — Best Linear Unbiased Estimator.
Finished this topic? Mark it so your progress, study plan and readiness keep up.
Stuck on something here?