The Mathematics Behind Transformers & LLMs

Overview

Large language models look intimidating, but their engine — self-attention — is built entirely from tools in this track. Each token becomes an embedding vector. From every token we compute three projections: a query, a key, and a value (three matrix multiplies). Attention scores how relevant every token is to every other token by taking dot products of queries with keys (the dot-product-as-similarity idea), scales them for numerical stability, and passes them through a softmax to get weights that form a probability distribution. Each token then gathers a weighted sum of value vectors — it literally attends to the tokens that matter. Stack these blocks with feed-forward layers and normalization, train the whole thing to minimise cross-entropy on next-token prediction, and you get an LLM. So the "magic" reduces to: embeddings (vectors), attention (dot products + softmax), and training (gradient descent on cross-entropy). Understanding this demystifies the most important architecture in AI.

Self-attention from scratch: Q, K, V

import numpy as np

def softmax(x):
    e = np.exp(x - x.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

seq, d = 3, 4                       # 3 tokens, dim 4
X = np.random.randn(seq, d)        # token embeddings
Wq, Wk, Wv = (np.random.randn(d, d) for _ in range(3))

Q, K, V = X @ Wq, X @ Wk, X @ Wv   # three projections
scores  = (Q @ K.T) / np.sqrt(d)   # dot-product relevance, scaled
weights = softmax(scores)          # rows sum to 1 -> a distribution
out     = weights @ V              # each token = weighted sum of values
print(weights.shape, out.shape)    # (3,3) attention map, (3,4) new vectors

Why every piece is math you already know

# Transformer component            -> Math topic in this track
# token embeddings                  -> vectors (Linear Algebra)
# Q, K, V projections               -> matrix multiplication
# attention scores  Q · K^T         -> dot product = similarity
# softmax over scores               -> categorical distribution (Probability)
# scale by 1/sqrt(d)                -> numerical stability
# next-token loss                   -> cross-entropy (Information Theory)
# training the weights              -> gradient descent + Adam (Calculus/Optim)
print("An LLM is these building blocks, scaled to billions of parameters.")

Key Points to Remember

• 1Tokens become embedding vectors; Q/K/V are three learned matrix projections
• 2Attention scores = scaled dot products (relevance), turned into weights by softmax
• 3Each token output is a weighted sum of value vectors — information mixing across tokens
• 4LLMs train by minimising cross-entropy on next-token prediction via gradient descent