The Mathematics Behind Neural Networks

Overview

This is where the whole track comes together. A neural network is just repeated application of two operations: a linear transformation (matrix multiply plus bias — linear algebra) and a non-linear activation (calculus-friendly functions like ReLU that let it fit curves). Data flows forward through the layers to produce a prediction; a loss function (information theory's cross-entropy or MSE) scores how wrong it is; then backpropagation (the chain rule) computes the gradient of that loss with respect to every weight, and an optimizer (gradient descent / Adam) nudges the weights downhill. Repeat over many mini-batches and the network learns. Nothing here is new — it is linear algebra for the forward pass, calculus for the gradients, probability for the outputs, and optimization for the updates, all working in concert. Seeing a network as this loop, rather than a black box, is exactly the understanding that makes you an AI engineer rather than an API caller.

Forward pass: alternate linear and non-linear

import numpy as np

def relu(x):    return np.maximum(0, x)
def softmax(x):
    e = np.exp(x - x.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)

X  = np.random.randn(4, 3)            # 4 examples, 3 features
W1, b1 = np.random.randn(3, 5), np.zeros(5)
W2, b2 = np.random.randn(5, 2), np.zeros(2)

H     = relu(X @ W1 + b1)             # hidden layer (linear + non-linear)
probs = softmax(H @ W2 + b2)          # output as a probability distribution
print(probs.shape, np.round(probs.sum(axis=1), 3))   # (4,2) rows sum to 1

The full training loop in miniature

import numpy as np

# 1-layer logistic regression trained by gradient descent
rng = np.random.default_rng(0)
X = rng.normal(size=(200, 2))
y = (X[:, 0] + X[:, 1] > 0).astype(float)     # true boundary
w, b, lr = np.zeros(2), 0.0, 0.1

for epoch in range(300):
    z = X @ w + b
    p = 1 / (1 + np.exp(-z))                   # forward
    grad_w = X.T @ (p - y) / len(y)            # backward (chain rule)
    grad_b = np.mean(p - y)
    w -= lr * grad_w; b -= lr * grad_b         # update
acc = np.mean((p > 0.5) == y)
print("accuracy:", round(acc, 3))              # ~0.9+ -> it learned

Key Points to Remember

• 1A network alternates linear transforms (XW+b) with non-linear activations
• 2Forward pass produces a prediction; a loss scores it (cross-entropy/MSE)
• 3Backpropagation = chain rule computing every weight's gradient in one backward sweep
• 4An optimizer (SGD/Adam) steps weights downhill; repeat over mini-batches to learn