Partial Derivatives & the Chain Rule

Overview

Neural networks are deeply nested functions: the loss depends on the last layer, which depends on the previous layer, and so on back to the weights. To train, you need the derivative of the loss with respect to each weight buried deep inside that nesting. Two ideas make this tractable. Partial derivatives let you ask "how does the loss change if I wiggle just this one weight?" while freezing everything else. The chain rule tells you how to differentiate a composition f(g(x)): multiply the outer slope by the inner slope. Applied layer by layer from the output backward, the chain rule becomes backpropagation — the algorithm that efficiently computes every weight's gradient in one backward sweep. Understanding "multiply the local slopes along the path" demystifies the single most important algorithm in deep learning.

Chain rule: multiply slopes through a composition

import numpy as np

# y = (3x + 1)^2 ; let u = 3x + 1, y = u^2
# dy/du = 2u , du/dx = 3 -> dy/dx = 2u * 3 = 6(3x+1)
def dy_dx(x):
    u = 3*x + 1
    return 2*u * 3

def y(x): return (3*x + 1)**2
# numerical check:
x, h = 2.0, 1e-6
print(round(dy_dx(x), 4))                       # 42.0
print(round((y(x+h) - y(x-h)) / (2*h), 4))      # 42.0

Backprop = the chain rule, run backward through layers

import numpy as np

# forward: h = w1 * x ; y = w2 * h ; loss = (y - target)^2
x, target = 2.0, 10.0
w1, w2 = 1.5, 2.0

h = w1 * x
y = w2 * h
loss = (y - target)**2

# backward (chain rule), layer by layer:
dloss_dy = 2 * (y - target)      # d loss / d y
dloss_dw2 = dloss_dy * h         # y = w2*h  -> * h
dloss_dh = dloss_dy * w2         # y = w2*h  -> * w2
dloss_dw1 = dloss_dh * x         # h = w1*x  -> * x
print(round(dloss_dw2, 3), round(dloss_dw1, 3))

Key Points to Remember

• 1A partial derivative varies one input while holding the others constant
• 2Chain rule: differentiate a composition by multiplying local slopes
• 3Backpropagation = the chain rule applied backward through the layers
• 4Autograd (PyTorch/JAX) automates this so you never hand-derive gradients