Optimization — Cheat Sheet

A loss function is the single number a model tries to minimise, and convexity describes whether that landscape is a simple bowl or a mountain range full of traps.

• ✓The loss function is the single number training minimises — it defines "good"
• ✓MSE for regression; cross-entropy for classification
• ✓Convex losses have one global minimum; gradient descent is guaranteed to reach it
• ✓Deep networks are non-convex — initialization, schedules, and momentum matter

import numpy as np

# Regression: mean squared error
y, yhat = np.array([3.0, 5.0, 2.0]), np.array([2.5, 5.0, 4.0])
mse = np.mean((y - yhat)**2)
print("MSE:", round(mse, 3))                       # 0.75

# Classification: binary cross-entropy
p = np.array([0.9, 0.1, 0.8]); t = np.array([1, 0, 1])
bce = -np.mean(t*np.log(p) + (1-t)*np.log(1-p))
print("BCE:", round(bce, 3))                       # low = confident & correct

Regularization adds a penalty for complexity to the loss so the model prefers simpler solutions — the main lever against overfitting, with L1 driving sparsity and L2 shrinking weights.

• ✓Regularization penalises complexity to reduce overfitting and improve generalization
• ✓L2 (weight decay): penalises Σw² — shrinks all weights, none exactly zero
• ✓L1: penalises Σ|w| — drives many weights to exactly zero (sparse, feature selection)
• ✓λ sets penalty strength; dropout, early stopping & augmentation are regularizers too

import numpy as np

w = np.array([2.0, -3.0, 0.1, 0.0, 1.5])
lam = 0.1

l2_penalty = lam * np.sum(w**2)      # weight decay: shrink all
l1_penalty = lam * np.sum(np.abs(w)) # sparsity: push many to 0
print("L2 penalty:", round(l2_penalty, 3))
print("L1 penalty:", round(l1_penalty, 3))
# total_loss = data_loss + penalty  -> optimizer balances fit vs simplicity

Modern optimizers speed up gradient descent by remembering past gradients (momentum) and adapting the step size per parameter (RMSProp) — Adam combines both and is the default for training deep nets.

• ✓Momentum accumulates a velocity from past gradients — faster, smoother descent
• ✓RMSProp adapts the step size per parameter using recent squared gradients
• ✓Adam combines momentum + adaptive scaling (+ bias correction) — the default optimizer
• ✓Understanding each mechanism lets you diagnose slow or unstable training

import numpy as np

def momentum_step(grad_fn, x, lr=0.1, beta=0.9, steps=20):
    v = 0.0
    for _ in range(steps):
        g = grad_fn(x)
        v = beta * v + (1 - beta) * g   # exponential average of gradients
        x -= lr * v                     # step along the velocity
    return round(x, 4)

# minimise f(x)=x^2, gradient 2x
print(momentum_step(lambda x: 2*x, x=5.0))   # -> near 0, faster than plain GD