Regression basics and the bias–variance trade-off
Basics of regression (linear least squares) and the bias–variance trade-off: definitions, OLS derivation, assumptions, error decomposition, diagrams to draw, worked numerical examples, common mistakes and exam tips.
Drafted with Aria, reviewed by the AiCanCode.org team · Updated . How we write and check our notes. Spotted an error? Use Give Feedback at the bottom of the page.
Why it matters
Regression is the basic tool engineers use when a continuous quantity (temperature, voltage, time, performance metric) must be predicted from one or more measured inputs. Understanding ordinary least squares (OLS) and the bias–variance trade-off helps you choose model complexity, avoid overfitting, and report reliable error estimates in tasks such as sensor calibration, performance modelling, and simple forecasting.
Key ideas
Define: Linear model (in parameters) — a model that is linear in parameters β: y_i = β_0 + β_1 x_i1 + ... + β_p x_ip + ε_i. Using matrix form: y = Xβ + ε.
Define: Ordinary least squares (OLS) — estimate β by minimising the sum of squared residuals SSE(β) = Σ_i (y_i − x_i^T β)^2.
Assumptions for OLS to be unbiased and have the standard properties (list as separate bullet points so an exam answer can tick them):
- Linearity: the data-generating process is y = Xβ + ε (linear in parameters).
- Full column rank: X^T X is invertible (no exact multicollinearity among predictors).
- Exogeneity: E[ε | X] = 0 (errors have zero mean conditional on X).
- Homoscedasticity: Var(ε_i | X) = σ^2 (constant variance of errors) — used for standard errors formula.
- No autocorrelation: Cov(ε_i, ε_j) = 0 for i ≠ j — needed in time-series or when observations are dependent.
- Normality of errors (optional, for finite-sample hypothesis tests): ε ~ N(0, σ^2 I).
Derivation (common exam derivation) of OLS closed form (matrix steps):
- Write SSE(β) = (y − Xβ)^T (y − Xβ).
- Differentiate with respect to β: ∂SSE/∂β = −2 X^T (y − Xβ).
- Set derivative = 0 (first-order condition): X^T y − X^T X β = 0.
- Solve normal equations: X^T X β = X^T y.
- If X^T X is invertible, β̂ = (X^T X)^{-1} X^T y.
Exams often ask to expand this for simple linear regression (p = 1). For scalar slope m and intercept b, derive formulas using sums:
- m = (n Σ x_i y_i − Σ x_i Σ y_i) / (n Σ x_i^2 − (Σ x_i)^2).
- b = (Σ y_i − m Σ x_i) / n.
Define: Residual r_i = y_i − ŷ_i (vertical distance; observed minus fitted). Draw a scatter plot of data points (x_i, y_i). Draw the fitted line ŷ = m x + b. For each point, draw a vertical line from the point to the line and label it r_i.
Define: Sum of squared errors (SSE) or residual sum of squares (RSS) = Σ r_i^2. Define Mean squared error (MSE) = (1/n) Σ r_i^2.
Define: Coefficient of determination (R^2) = 1 − RSS / TSS, where TSS = Σ (y_i − ȳ)^2.
Define: Bias of an estimator f̂(x): Bias[f̂(x)] = E[f̂(x)] − f(x) (difference between average model prediction and true function at x).
Define: Variance of an estimator f̂(x): Var[f̂(x)] = E[(f̂(x) − E[f̂(x)])^2].
Derive the bias–variance decomposition (standard exam derivation) for squared error at a fixed input x_0. Suppose y = f(x_0) + ε with E[ε] = 0 and Var(ε) = σ^2. Then:
- Start with expected squared prediction error: E[(y − f̂(x_0))^2] = E[(f(x_0) + ε − f̂(x_0))^2].
- Expand: = E[(f(x_0) − f̂(x_0))^2] + 2 E[ε (f(x_0) − f̂(x_0))] + E[ε^2]. The cross-term is zero because E[ε] = 0 and ε independent of training set (or zero-mean conditional on X).
- So = E[(f̂(x_0) − f(x_0))^2] + σ^2.
- Decompose E[(f̂ − f)^2] = (E[f̂] − f)^2 + E[(f̂ − E[f̂])^2] = Bias^2 + Variance.
- Final result (pointwise): E[(y − f̂(x_0))^2] = σ^2 + Bias[f̂(x_0)]^2 + Var[f̂(x_0)].
Comment: σ^2 is the irreducible error (noise) you cannot remove by any model.
Explain trade-off: Increasing model complexity usually reduces bias (model can fit training pattern better) but increases variance (sensitive to training set), while simpler models have higher bias and lower variance. The optimal complexity minimises total expected error.
Diagram to draw for bias–variance: draw a horizontal axis of model complexity (degree or number of parameters) and vertical axis expected error. Plot three curves: bias^2 decreasing, variance increasing, and total error forming a U-shaped curve. Label the minimum point.
Formulas
y = X β + ε
- y: vector of target outputs (SI: units of predicted quantity, e.g., V for voltage). X: matrix of input features (each column a predictor, SI: units depend on feature). β: parameter vector (units scale accordingly). ε: error vector (same units as y). Applies to linear models.
SSE(β) = (y − X β)^T (y − X β)
- SSE: sum of squared errors (units: (unit of y)^2). Applies when using least squares objective.
X^T X β = X^T y
- Normal equations. Applies under OLS derivation; requires full column rank for unique solution.
β̂ = (X^T X)^{-1} X^T y
- β̂: OLS estimate of β. Applies when X^T X is invertible. Under the Gauss–Markov conditions (E[ε|X] = 0 and Var(ε|X) = σ^2 I), β̂ is the BLUE. If X^T X is singular, use the Moore–Penrose pseudoinverse: β̂ = X^+ y.
m = (n Σ x_i y_i − Σ x_i Σ y_i) / (n Σ x_i^2 − (Σ x_i)^2)
- m: slope for simple linear regression. x_i, y_i: data. n: number of observations.
b = (Σ y_i − m Σ x_i) / n
- b: intercept in simple linear regression.
RSS = Σ (y_i − ŷ_i)^2
- Residual sum of squares. Units: (unit of y)^2.
MSE = (1/n) Σ (y_i − ŷ_i)^2
- Mean squared error. Units: (unit of y)^2. Use for average error measurement. In linear regression, an unbiased estimator of the noise variance is s^2 = RSS / (n − p − 1) (with an intercept and p predictors); Var(β̂ | X) = s^2 (X^T X)^{-1} and standard errors are the square roots of the diagonal entries.
R^2 = 1 − RSS / TSS, where TSS = Σ (y_i − ȳ)^2
- R^2: fraction of variance explained (unitless). With an intercept term, 0 ≤ R^2 ≤ 1; without an intercept it can be negative. Closer to 1 is better in-sample fit. Adjusted R^2 = 1 − (RSS/(n − p − 1)) / (TSS/(n − 1)).
E[(y − f̂(x))^2] = σ^2 + Bias[f̂(x)]^2 + Var[f̂(x)]
- Pointwise bias–variance decomposition. σ^2: irreducible noise variance (units: (unit of y)^2).
Worked examples
Example 1 — Exact linear data (simple linear regression) Given data (x in m, y in m): (0, 1), (1, 3), (2, 5). Find slope m and intercept b using formulas and show fitted line. Units: metres (m). Steps:
- n = 3, Σ x = 0 + 1 + 2 = 3, Σ y = 1 + 3 + 5 = 9.
- Σ x y = 0·1 + 1·3 + 2·5 = 0 + 3 + 10 = 13.
- Σ x^2 = 0^2 + 1^2 + 2^2 = 5.
- Slope: m = (n Σ x y − Σ x Σ y) / (n Σ x^2 − (Σ x)^2) = (3·13 − 3·9) / (3·5 − 3^2) = (39 − 27) / (15 − 9) = 12 / 6 = 2 (unit: m/m → unitless).
- Intercept: b = (Σ y − m Σ x)/n = (9 − 2·3) / 3 = (9 − 6) / 3 = 1 m.
- Fitted line: ŷ = 2 x + 1. Residuals are zero; SSE = 0. Final answer: m = 2, b = 1; ŷ = 2 x + 1 (x and y in m).
Example 2 — Noisy linear data (simple linear regression) Given data (x in m, y in m): (0, 1.2), (1, 2.9), (2, 4.8). Compute m, b, predictions and SSE. Steps:
- n = 3, Σ x = 3, Σ y = 1.2 + 2.9 + 4.8 = 8.9.
- Σ x y = 0·1.2 + 1·2.9 + 2·4.8 = 0 + 2.9 + 9.6 = 12.5.
- Σ x^2 = 5.
- m = (3·12.5 − 3·8.9) / (15 − 9) = (37.5 − 26.7) / 6 = 10.8 / 6 = 1.8 (unitless).
- b = (8.9 − 1.8·3) / 3 = (8.9 − 5.4) / 3 = 3.5 / 3 = 1.1666667 m.
- Predictions: ŷ(0) = 1.1666667, ŷ(1) = 2.9666667, ŷ(2) = 4.7666667 (m).
- Residuals: r_1 = 1.2 − 1.1666667 = 0.0333333, r_2 = 2.9 − 2.9666667 = −0.0666667, r_3 = 4.8 − 4.7666667 = 0.0333333.
- SSE = (0.0333333)^2 + (−0.0666667)^2 + (0.0333333)^2 = 0.0011111 + 0.0044444 + 0.0011111 = 0.0066667 (m^2). Final answer: m = 1.8, b ≈ 1.1666667, SSE ≈ 0.0066667 m^2.
Example 3 — Bias–variance model-answer (theory question) Question: Show the bias–variance decomposition for expected squared error at x_0 and explain terms. Model answer: (Give the derivation as in Key ideas above.) Conclude with interpretation: σ^2 is irreducible; Bias^2 is error from systematic mismatch; Variance is sensitivity to training data. Models that reduce bias (more flexible) often increase variance; choose complexity to minimise total.
Common mistakes
- Using the formula β̂ = (X^T X)^{-1} X^T y even when X^T X is singular (forgetting to state full-rank requirement).
- Confusing residuals r_i = y_i − ŷ_i with parameter errors β̂ − β.
- Mixing up RSS and MSE and misreporting units (RSS has squared units, MSE average squared units).
- In bias–variance statements: saying "more complex always worse" or ignoring irreducible error σ^2.
- Arithmetic mistakes in hand calculations (forget to carry units or wrong summation). Recompute sums twice.
For your exam
- Derivation questions: derive normal equations and β̂ = (X^T X)^{-1} X^T y. Full marks: show SSE, differentiation, set to zero, solve, mention invertibility assumption.
- Numerical questions: compute m and b from small datasets. Full marks: show all sums, substitute into formulas, compute residuals and SSE with units.
- Theory questions: derive bias–variance decomposition. Full marks: stepwise algebra, explanation of cross-term cancellation, interpretation.
- Diagram questions: draw scatter with fitted line and residuals; draw bias/variance vs complexity plot. Full marks: axes labelled, curves labelled, minimum point marked.
- Short-note: describe how to pick model complexity using cross-validation. Full marks: mention variance, bias, and cross-validation as practical tool; refer to later topic Cross-validation.
Quick check
- What is the normal equation for OLS? (write in matrix form)
- State the bias–variance decomposition for pointwise squared error.
- For simple linear regression, what is the unit of slope m if x is in seconds and y is in metres?
Answers: 1) X^T X β = X^T y. 2) E[(y − f̂(x))^2] = σ^2 + Bias[f̂(x)]^2 + Var[f̂(x)]. 3) metres per second (m/s).
Finished this topic? Mark it so your progress, study plan and readiness keep up.
Stuck on something here?