Linear Algebra — Cheat Sheet

Scalars, vectors, matrices and tensors are just numbers arranged in 0, 1, 2, and N dimensions — the containers that hold every piece of data an AI model ever sees.

• ✓Scalar (0-D), vector (1-D), matrix (2-D), tensor (N-D) — numbers in a grid
• ✓A vector = one example or embedding; a matrix = a batch or a weight layer
• ✓.shape reads left-to-right as the meaning of each axis (batch, features, …)
• ✓Most ML bugs are shape mismatches — print shapes constantly

import numpy as np

scalar = np.array(7.0)                       # 0-D
vector = np.array([1.0, 2.0, 3.0])           # 1-D: one example / embedding
matrix = np.array([[1, 2, 3], [4, 5, 6]])    # 2-D: 2 examples x 3 features
tensor = np.zeros((32, 28, 28, 3))           # 4-D: batch of 32 RGB images

for a in (scalar, vector, matrix, tensor):
    print(a.ndim, a.shape)
# 0 ()            1 (3,)        2 (2, 3)      4 (32, 28, 28, 3)

The dot product multiplies two vectors into a single number that measures how much they point the same way — the core operation behind similarity search, attention, and every neuron.

• ✓Dot product: multiply matching entries and sum into one number
• ✓It measures alignment — large & positive = same direction, 0 = perpendicular
• ✓A neuron is a dot product of inputs and weights (plus a bias)
• ✓Cosine similarity = normalised dot product; the basis of embedding/RAG search

import numpy as np

a = np.array([1.0, 2.0, 3.0])
b = np.array([4.0, 5.0, 6.0])

# manual: 1*4 + 2*5 + 3*6 = 32
print(np.sum(a * b))    # 32.0
print(a @ b)            # 32.0  -> the @ operator is the dot product

Matrix multiplication is a grid of dot products that transforms a whole batch of vectors at once — it is the single most executed operation in all of deep learning.

• ✓Matrix multiply = a grid of dot products; result[i,j] = row i · column j
• ✓Shape rule: (m×k)·(k×n) → (m×n); inner dimensions must match
• ✓A dense layer is exactly one matmul plus a bias: H = XW + b
• ✓GPUs accelerate AI because they parallelise large matrix multiplications

import numpy as np

X = np.random.randn(32, 784)    # 32 examples, 784 features (m x k)
W = np.random.randn(784, 128)   # weight matrix (k x n)

H = X @ W                       # (32, 784) @ (784, 128)
print(H.shape)                  # (32, 128)  -> 32 examples, 128 hidden units

# Mismatch -> error:
# X @ np.random.randn(100, 128)  # ValueError: 784 != 100

Transpose flips a matrix over its diagonal, the identity matrix is the "1" of matrix math, and the inverse is the "undo" operation — together they let you reshape data and solve linear systems.

• ✓Transpose (.T) swaps rows/columns — used to align shapes and compute Q·Kᵀ
• ✓Identity matrix I is the "1": AI = IA = A
• ✓Inverse A⁻¹ undoes A (A·A⁻¹ = I) and solves Ax = b in closed form
• ✓Singular (non-invertible) matrices signal redundant/collinear data; prefer solve() over inv()

import numpy as np

A = np.array([[1, 2, 3],
              [4, 5, 6]])      # (2, 3)
print(A.T.shape)              # (3, 2)

# Attention scores need queries · keys^T:
Q = np.random.randn(4, 8)     # 4 tokens, 8-dim
K = np.random.randn(4, 8)
scores = Q @ K.T              # (4, 8) @ (8, 4) -> (4, 4)
print(scores.shape)           # (4, 4): every token scored against every token

A matrix is not just a grid of numbers — it is a function that rotates, scales, and shears space, and seeing it that way makes every neural-net layer intuitive.

• ✓A matrix is a function that transforms space (rotate, scale, shear, flip)
• ✓The columns of a matrix show where the basis (axis) vectors land
• ✓A neural layer XW relocates data into a new coordinate system
• ✓"Representation learning" = learning transformations that make data separable

import numpy as np

# 90-degree rotation matrix
theta = np.pi / 2
R = np.array([[np.cos(theta), -np.sin(theta)],
              [np.sin(theta),  np.cos(theta)]])

e1 = np.array([1.0, 0.0])     # x-axis unit vector
e2 = np.array([0.0, 1.0])     # y-axis unit vector
print(np.round(R @ e1, 3))    # [0. 1.]  -> x-axis lands on y-axis
print(np.round(R @ e2, 3))    # [-1. 0.] -> y-axis lands on -x-axis

Eigenvectors are the special directions a matrix does not rotate — it only stretches them — and the eigenvalue is how much; they reveal the "principal axes" of data behind PCA and stability analysis.

• ✓Eigenvector: a direction a matrix scales but does not rotate (Av = λv)
• ✓Eigenvalue λ: the scaling factor along that direction
• ✓PCA uses the top eigenvectors of the covariance for dimensionality reduction
• ✓Eigenvalues indicate stability — relevant to exploding/vanishing gradients

import numpy as np

A = np.array([[2.0, 0.0],
              [0.0, 3.0]])
vals, vecs = np.linalg.eig(A)
print(vals)                 # [2. 3.]  -> eigenvalues
print(vecs)                 # columns are eigenvectors (here the x and y axes)

v = vecs[:, 1]              # eigenvector for lambda = 3
print(np.allclose(A @ v, 3.0 * v))   # True -> Av = λv

SVD factors ANY matrix into rotate → scale → rotate (U Σ Vᵀ), giving you the single most useful tool for compression, noise removal, recommendations, and low-rank model tricks like LoRA.

• ✓SVD factors ANY matrix: M = U Σ Vᵀ (rotate → scale → rotate)
• ✓Singular values rank directions by importance (energy/variance)
• ✓Keeping the top-k singular values gives the best rank-k approximation
• ✓Powers compression, denoising, latent-factor recommenders, and LoRA fine-tuning

import numpy as np

M = np.random.randn(6, 4)
U, S, Vt = np.linalg.svd(M, full_matrices=False)
print(U.shape, S.shape, Vt.shape)   # (6,4) (4,) (4,4)
print(np.round(S, 2))               # singular values, largest first

# Reconstruct exactly:
M2 = U @ np.diag(S) @ Vt
print(np.allclose(M, M2))           # True