Calculus — Cheat Sheet

A limit asks "what value is this function heading toward?" and continuity means "no sudden jumps" — the smoothness that lets gradient-based learning work at all.

• ✓A limit is the value a function approaches as the input nears a point
• ✓A derivative is defined as a limit of slopes over a shrinking interval
• ✓Continuity = no jumps; small input change → small output change
• ✓Networks use smooth activations so gradient descent can follow the loss downhill

import numpy as np

def f(x):
    return np.sin(x) / x        # undefined exactly at 0

for x in [0.1, 0.01, 0.001]:
    print(x, round(f(x), 6))
# heads toward 1 as x -> 0, even though f(0) is 0/0
# 0.1   0.998334
# 0.01  0.999983
# 0.001 1.0

A derivative is the slope of a function — how fast the output changes as you nudge the input — and the gradient bundles those slopes for many inputs into the single "uphill" direction that training follows downhill.

• ✓A derivative is the slope: how the output changes per unit change of input
• ✓Zero derivative = flat point (min, max, or saddle)
• ✓The gradient is the vector of partial derivatives for all inputs
• ✓The gradient points uphill; training steps along the negative gradient

import numpy as np

def f(x):  return x**2
def numerical_deriv(f, x, h=1e-6):
    return (f(x + h) - f(x - h)) / (2 * h)    # slope over a tiny interval

print(round(numerical_deriv(f, 3.0), 4))      # 6.0  (analytic: 2x = 6)
print(round(numerical_deriv(f, -2.0), 4))     # -4.0 (2x = -4)

A partial derivative isolates one input's effect while holding others fixed, and the chain rule multiplies slopes through composed functions — together they ARE backpropagation.

• ✓A partial derivative varies one input while holding the others constant
• ✓Chain rule: differentiate a composition by multiplying local slopes
• ✓Backpropagation = the chain rule applied backward through the layers
• ✓Autograd (PyTorch/JAX) automates this so you never hand-derive gradients

import numpy as np

# y = (3x + 1)^2 ; let u = 3x + 1, y = u^2
# dy/du = 2u , du/dx = 3 -> dy/dx = 2u * 3 = 6(3x+1)
def dy_dx(x):
    u = 3*x + 1
    return 2*u * 3

def y(x): return (3*x + 1)**2
# numerical check:
x, h = 2.0, 1e-6
print(round(dy_dx(x), 4))                       # 42.0
print(round((y(x+h) - y(x-h)) / (2*h), 4))      # 42.0

Gradient descent is the loop that actually trains models: compute the gradient of the loss, take a small step downhill, repeat — with the learning rate controlling the step size.

• ✓Gradient descent: loss → gradient → step downhill → repeat
• ✓Learning rate sets step size: too small crawls, too large diverges
• ✓Batch (all data), stochastic (one), mini-batch (a few dozen — the default)
• ✓SGD, Adam, etc. are gradient descent with smarter step rules

import numpy as np

X = np.array([1.0, 2.0, 3.0, 4.0])
Y = 2.0 * X + 1.0                 # true w=2, b=1

w, b, lr = 0.0, 0.0, 0.01
for step in range(1000):
    pred = w * X + b
    error = pred - Y
    grad_w = np.mean(2 * error * X)   # d MSE / d w
    grad_b = np.mean(2 * error)       # d MSE / d b
    w -= lr * grad_w                  # step downhill
    b -= lr * grad_b

print(round(w, 2), round(b, 2))       # ~2.0 ~1.0 -> it learned the line