Derivatives & Gradients

Overview

The derivative answers one question: if I increase this input a little, how much does the output change, and in which direction? Positive slope means "output rises", negative means "output falls", zero means "flat — a peak, valley, or plateau". When a function has many inputs (a model has millions of weights), each has its own partial slope, and stacking them into a vector gives the gradient. The gradient points in the direction of steepest increase, so its negative points straight downhill — which is the entire strategy of training: compute the gradient of the loss with respect to every weight, then step the weights in the opposite direction to reduce the loss. Everything from linear regression to GPT is trained this way.

Derivative = slope = sensitivity of output to input

import numpy as np

def f(x):  return x**2
def numerical_deriv(f, x, h=1e-6):
    return (f(x + h) - f(x - h)) / (2 * h)    # slope over a tiny interval

print(round(numerical_deriv(f, 3.0), 4))      # 6.0  (analytic: 2x = 6)
print(round(numerical_deriv(f, -2.0), 4))     # -4.0 (2x = -4)

Gradient: the vector of slopes, pointing uphill

import numpy as np

# f(x, y) = x^2 + y^2 ; gradient = [2x, 2y]
def grad(x, y):
    return np.array([2*x, 2*y])

point = (3.0, 4.0)
g = grad(*point)
print(g)                    # [6. 8.] -> steepest-uphill direction
print(-g)                   # [-6. -8.] -> step this way to go downhill

Key Points to Remember

• 1A derivative is the slope: how the output changes per unit change of input
• 2Zero derivative = flat point (min, max, or saddle)
• 3The gradient is the vector of partial derivatives for all inputs
• 4The gradient points uphill; training steps along the negative gradient