Gradient Descent

Overview

Gradient descent is the algorithm that turns "the math of slopes" into a model that learns. You start with random weights, compute the loss and its gradient (which way is uphill), and nudge the weights a little in the opposite direction. Repeat thousands of times and the loss walks downhill toward a minimum. The learning rate is the crucial knob: too small and training crawls; too large and it overshoots and diverges. Variants matter in practice — batch gradient descent uses all data per step (accurate but slow), stochastic uses one example (noisy but fast), and mini-batch (the default) uses a few dozen to few hundred for the best of both. Every optimizer you will meet (SGD, Adam) is gradient descent with smarter step rules layered on top.

The core loop: step against the gradient

import numpy as np

X = np.array([1.0, 2.0, 3.0, 4.0])
Y = 2.0 * X + 1.0                 # true w=2, b=1

w, b, lr = 0.0, 0.0, 0.01
for step in range(1000):
    pred = w * X + b
    error = pred - Y
    grad_w = np.mean(2 * error * X)   # d MSE / d w
    grad_b = np.mean(2 * error)       # d MSE / d b
    w -= lr * grad_w                  # step downhill
    b -= lr * grad_b

print(round(w, 2), round(b, 2))       # ~2.0 ~1.0 -> it learned the line

Learning rate: the make-or-break knob

import numpy as np

def descend(lr, steps=8):
    x = 5.0                       # minimise f(x)=x^2, min at 0
    for _ in range(steps):
        x -= lr * (2 * x)         # step against gradient 2x
    return round(x, 3)

print("lr=0.1 :", descend(0.1))   # -> heads toward 0 (good)
print("lr=1.1 :", descend(1.1))   # -> diverges/explodes (too big)

Key Points to Remember

• 1Gradient descent: loss → gradient → step downhill → repeat
• 2Learning rate sets step size: too small crawls, too large diverges
• 3Batch (all data), stochastic (one), mini-batch (a few dozen — the default)
• 4SGD, Adam, etc. are gradient descent with smarter step rules