Regularisation: ridge and lasso
Ridge and lasso are L2 and L1 regularisation methods for linear models that reduce overfitting by penalising large coefficients; ridge shrinks coefficients continuously, lasso can set some to zero and perform feature selection.
Drafted with Aria, reviewed by the AiCanCode.org team · Updated . How we write and check our notes. Spotted an error? Use Give Feedback at the bottom of the page.
Why it matters
Regularisation is used when a linear model overfits noisy data or when you have many correlated features. Ridge (L2) and lasso (L1) add penalty terms to the regression objective so an engineer gets more stable predictions, better generalisation, and (for lasso) sparse models useful in limited-hardware deployments.
Key ideas
Start from the ordinary least squares (OLS) linear regression problem. Let X be the n×p design matrix, y the n×1 response vector, and β the p×1 coefficient vector. OLS minimises the residual sum of squares (RSS): RSS(β) = ||y − Xβ||^2. When p is large, X^T X ill-conditioned, or noise is high, OLS has high variance and may overfit.
Definition: Regularisation is adding a penalty term to the loss to control model complexity. Equivalent constrained view: the penalised problems min ||y − Xβ||^2 + λ||β||_2^2 (ridge) and min ||y − Xβ||^2 + λ||β||_1 (lasso) are equivalent to constrained forms min ||y − Xβ||^2 subject to ||β||_2^2 ≤ t or ||β||_1 ≤ t, respectively; for each λ ≥ 0 there exists t ≥ 0 yielding the same solution set, and vice versa.
Definition: Ridge regression (also called Tikhonov regularisation) minimises RSS plus an L2 penalty: minimise ||y − Xβ||^2 + λ||β||2^2 where λ ≥ 0 is the regularisation hyperparameter. Assumptions: standard linear model (no special distribution needed for estimator), and typically we centre y and standardise X so all features are on comparable scale; the intercept is normally not penalised.
Derivation of the ridge closed-form solution (unpenalised intercept assumed zero or data centred):
- Write the objective J(β) = (y − Xβ)^T (y − Xβ) + λβ^T β.
- Differentiate w.r.t. β: ∂J/∂β = −2X^T(y − Xβ) + 2λβ.
- Set gradient to zero: X^T X β + λβ = X^T y.
- Rearranged: (X^T X + λI) β = X^T y.
- If (X^T X + λI) is invertible (it is for λ > 0), β_ridge = (X^T X + λI)^{-1} X^T y.
Remarks: Ridge shrinks coefficients towards zero but does not set exactly to zero (except in degenerate limits). Geometrically, ridge minimisation is intersection of RSS contours (ellipses) with a circular L2 ball; solution lies where an ellipse first touches the circle.
Definition: Lasso regression minimises RSS plus an L1 penalty: minimise ||y − Xβ||^2 + λ||β||1. Properties: the objective is convex but not differentiable where any β_j = 0. The non-differentiability allows some coefficients to be exactly zero => feature selection.
Subgradient condition (first-order optimality) for lasso: 0 ∈ −2X^T(y − Xβ) + λ s where s_j ∈ sign(β_j) for β_j ≠ 0 and s_j ∈ [−1, 1] for β_j = 0.
No general closed-form solution exists for lasso. Common solvers: coordinate descent, least-angle regression (LARS) for the full path, and convex optimisation (quadratic programming). For special cases (orthonormal columns) the lasso has the elementwise soft-thresholding closed form.
Bayesian interpretation: ridge corresponds to a Gaussian prior β_j ∼ N(0, τ^2) (equivalently penalising squared coefficients), lasso corresponds to a Laplace (double exponential) prior p(β_j) ∝ exp(−α|β_j|). This explains shrinkage behaviour: Gaussian concentrates mass around zero but smooth; Laplace has sharper peak and heavier tails causing exact zeros.
Practical notes and assumptions:
- Always standardise (zero mean, unit variance) features before applying ridge or lasso (unless you have good reason), because penalties depend on scale.
- Do not penalise the intercept; centre y and X if you want to include intercept separately.
- Choose λ by cross-validation (k-fold CV) to trade bias and variance.
- Lasso can be unstable when many correlated features exist; elastic net (combination of L1 and L2) helps. Elastic net objective: J_enet(β) = ||y − Xβ||^2 + λ1 ||β||_1 + λ2 ||β||_2^2; choose λ1, λ2 by cross-validation.
Geometric picture to draw for an exam: a 2D plot with axes β1 and β2. Draw concentric RSS ellipses centred at the OLS solution and draw:
- a circular constraint region for ridge (L2 ball); label it "L2 ball (ridge)".
- a diamond-shaped region for lasso (L1 ball); label it "L1 ball (lasso)". Show the point where the smallest ellipse touches each region: ridge touch on side (small nonzero β1, β2), lasso often touches at a corner (one coefficient zero). Label the OLS point and the ridge/lasso solutions.
Formulas
J_OLS(β) = ||y − Xβ||^2 — RSS to minimise. y: response (SI units of measured output), X: design matrix (each column has units of its feature), β: coefficients (units output per unit input), applies to linear regression.
J_ridge(β) = ||y − Xβ||^2 + λ ||β||2^2 — ridge objective; λ ≥ 0 is penalty parameter (unit: squared coefficient scaling, dimensionless in practice because data standardised).
β_ridge = (X^T X + λ I)^{-1} X^T y — closed-form ridge estimator; applies when X is centred (no intercept) or intercept not penalised; requires λ > 0 for guaranteed invertibility.
J_lasso(β) = ||y − Xβ||^2 + λ ||β||1 — lasso objective; convex but nondifferentiable where β_j = 0.
0 ∈ −2 X^T (y − Xβ) + λ s where s_j ∈ sign(β_j) (if β_j ≠ 0) or s_j ∈ [−1,1] (if β_j = 0) — subgradient optimality for lasso.
(Orthonormal columns case: X^T X = I)
β_j^OLS = X_j^T y — OLS coefficient for column j.
β_j^ridge = β_j^OLS / (1 + λ) — ridge shrinkage for orthonormal X.
β_j^lasso = S(β_j^OLS, λ/2) = sign(β_j^OLS) max(|β_j^OLS| − λ/2, 0) — lasso soft-threshold for orthonormal X under the objective defined above (no 1/2 factor on RSS).
df_ridge(λ) = trace( X (X^T X + λ I)^{-1} X^T ) — effective degrees of freedom for ridge.
Cross-validation selection: choose λ that minimises CV estimate of prediction error (e.g., mean squared error on held-out folds).
Worked examples
Example 1 — ridge with small numeric data (centred design, identity X for clarity) Problem: n = p = 2, X = I (2×2 identity), y = [3, −1]^T (units: metres). Use ridge penalty λ = 2. Objective J = ||y − Xβ||^2 + λ||β||2^2. Compute β_ridge. Steps:
- X^T X = I. Compute (X^T X + λ I) = I + 2 I = 3 I.
- Invert: (3 I)^{-1} = (1/3) I.
- Compute X^T y = y = [3, −1]^T.
- β_ridge = (1/3) y = [1, −1/3]^T (units: metres per unit feature; since X=I, numerically metres). Final answer: β_ridge = [1.000, −0.333] (metres).
Example 2 — lasso soft-threshold (orthonormal columns) Problem: same X = I, y = [3, −1]^T, λ = 2. Objective J = ||y − Xβ||^2 + λ||β||1 with standard scaling leading to soft-thresholding. Steps:
- β^OLS = X^T y = y = [3, −1].
- Apply soft-threshold S(·, λ/2): β1 = sign(3)·max(3 − 1, 0) = 2. β2 = sign(−1)·max(1 − 1, 0) = 0. Final answer: β_lasso = [2.000, 0.000] — lasso sets the second coefficient exactly to zero under this scaling.
Example 3 — short model answer (derivation question) Question: "Derive the ridge normal equations and state conditions for invertibility." Model answer: Show objective, compute gradient, set to zero, give (X^T X + λI) β = X^T y, state invertible if λ > 0 or X has full column rank. Mention intercept centring.
Common mistakes
- Confusing λ scaling conventions: some books put (1/2n) factor in front of RSS or use λ/n; this changes the numerical value of λ and the soft-threshold constant. State your objective definition before computing.
- Expecting lasso always to pick the correct features; with highly correlated predictors it may choose one arbitrarily.
- Using closed-form ridge formula when λ is 0 and X^T X is singular — then inverse does not exist.
- Mis-drawing geometry: L1 ball is a diamond (4-corner) in 2D, L2 ball is circular.
For your exam
Typical question types:
- Definitions: give precise definitions of ridge and lasso objectives; state properties (convexity, differentiability).
- Short notes: compare ridge vs lasso; discuss when to use each and practical issues (standardisation, cross-validation).
- Derivations: derive ridge closed-form and show why lasso lacks closed form; derive soft-threshold for orthonormal X.
- Numericals: compute β_ridge given a small X,y and λ; compute β_lasso in orthonormal case.
- Diagrams: draw geometric interpretation in β1–β2 plane and explain. Full-marks answers include: clear statement of objective, assumptions (centred data, penalty not applied to intercept), step-by-step derivation or computation, units where applicable, and clear concluding statement about properties (shrinkage vs sparsity).
Quick check
- What is the ridge estimator formula for centred data?
- Why does lasso produce sparse solutions whereas ridge does not?
- Name one practical step you must do before applying ridge or lasso to raw data. Answers: 1) β_ridge = (X^T X + λ I)^{-1} X^T y. 2) Because the L1 penalty is nondifferentiable at zero and geometry of L1 ball encourages corner solutions with some β_j = 0. 3) Standardise (zero mean, unit variance) the features.
Finished this topic? Mark it so your progress, study plan and readiness keep up.
Stuck on something here?