Momentum, RMSProp & Adam

Overview

Plain gradient descent takes a fixed step along the current gradient, which is slow in ravines and jittery with noisy mini-batches. Momentum fixes this by accumulating a velocity — an exponentially decaying average of past gradients — so the optimizer builds speed in consistent directions and damps oscillation, like a ball rolling downhill. RMSProp adapts the learning rate for each parameter individually by dividing by a running average of recent squared gradients, so steep directions get smaller steps and flat directions get larger ones. Adam ("Adaptive Moment estimation") combines both: momentum for direction plus per-parameter adaptive scaling, with a bias correction for the early steps. This is why Adam is the near-universal default for deep learning — it trains fast and needs little learning-rate babysitting. Knowing what each mechanism adds lets you reason about training dynamics instead of guessing.

Momentum: accumulate velocity to move faster

import numpy as np

def momentum_step(grad_fn, x, lr=0.1, beta=0.9, steps=20):
    v = 0.0
    for _ in range(steps):
        g = grad_fn(x)
        v = beta * v + (1 - beta) * g   # exponential average of gradients
        x -= lr * v                     # step along the velocity
    return round(x, 4)

# minimise f(x)=x^2, gradient 2x
print(momentum_step(lambda x: 2*x, x=5.0))   # -> near 0, faster than plain GD

Adam: momentum + per-parameter adaptive steps

import numpy as np

def adam_step(grad_fn, x, lr=0.1, b1=0.9, b2=0.999, eps=1e-8, steps=50):
    m = v = 0.0
    for t in range(1, steps + 1):
        g = grad_fn(x)
        m = b1*m + (1-b1)*g            # 1st moment (momentum)
        v = b2*v + (1-b2)*g*g          # 2nd moment (adaptive scale)
        mhat = m / (1 - b1**t)         # bias correction
        vhat = v / (1 - b2**t)
        x -= lr * mhat / (np.sqrt(vhat) + eps)
    return round(x, 4)

print(adam_step(lambda x: 2*x, x=5.0))   # -> near 0

Key Points to Remember

• 1Momentum accumulates a velocity from past gradients — faster, smoother descent
• 2RMSProp adapts the step size per parameter using recent squared gradients
• 3Adam combines momentum + adaptive scaling (+ bias correction) — the default optimizer
• 4Understanding each mechanism lets you diagnose slow or unstable training