Regularization (L1 & L2)

Overview

A model that memorises its training data but fails on new data has overfit. Regularization fights this by adding a penalty on the size of the weights to the loss, nudging the optimizer toward simpler, smoother functions that generalise better. L2 regularization (weight decay) penalises the sum of squared weights, shrinking them all toward zero without forcing any to be exactly zero — it discourages any single weight from dominating and is the default in deep learning. L1 penalises the sum of absolute weights, which tends to push many weights exactly to zero, performing automatic feature selection and yielding sparse models. The strength of the penalty (λ) trades off fitting the data against staying simple. Related tools — dropout, early stopping, data augmentation — are regularizers too. This is arguably the most practically important optimization concept for shipping models that work in the real world.

L2 shrinks all weights; L1 zeros many out

import numpy as np

w = np.array([2.0, -3.0, 0.1, 0.0, 1.5])
lam = 0.1

l2_penalty = lam * np.sum(w**2)      # weight decay: shrink all
l1_penalty = lam * np.sum(np.abs(w)) # sparsity: push many to 0
print("L2 penalty:", round(l2_penalty, 3))
print("L1 penalty:", round(l1_penalty, 3))
# total_loss = data_loss + penalty  -> optimizer balances fit vs simplicity

The λ knob: underfit ↔ overfit

# Conceptual sweep of regularization strength:
for lam in [0.0, 0.01, 0.1, 10.0]:
    if lam == 0.0:      note = "no penalty  -> risk OVERFIT"
    elif lam <= 0.1:    note = "balanced     -> good generalization"
    else:               note = "huge penalty -> risk UNDERFIT"
    print(f"lambda={lam:<5} {note}")

Key Points to Remember

• 1Regularization penalises complexity to reduce overfitting and improve generalization
• 2L2 (weight decay): penalises Σw² — shrinks all weights, none exactly zero
• 3L1: penalises Σ|w| — drives many weights to exactly zero (sparse, feature selection)
• 4λ sets penalty strength; dropout, early stopping & augmentation are regularizers too