Singular Value Decomposition (SVD)

Overview

Singular Value Decomposition is the crown jewel of applied linear algebra because it works on every matrix (unlike eigen-decomposition, which needs square matrices). It breaks any matrix M into three pieces: M = U Σ Vᵀ — a rotation (V ᵀ), a scaling by the singular values (Σ), and another rotation (U). The singular values, sorted large to small, tell you how much "energy" each direction carries; keeping only the top few gives the best possible low-rank approximation of the matrix. That one fact underlies image compression, latent-factor recommendation systems (decomposing a user×item matrix), noise reduction, and — very topically — LoRA, the technique that fine-tunes giant LLMs cheaply by learning tiny low-rank update matrices. If you understand SVD as "find the most important directions and throw the rest away", you understand a huge swath of practical ML.

M = U Σ Vᵀ: any matrix as rotate-scale-rotate

import numpy as np

M = np.random.randn(6, 4)
U, S, Vt = np.linalg.svd(M, full_matrices=False)
print(U.shape, S.shape, Vt.shape)   # (6,4) (4,) (4,4)
print(np.round(S, 2))               # singular values, largest first

# Reconstruct exactly:
M2 = U @ np.diag(S) @ Vt
print(np.allclose(M, M2))           # True

Low-rank approximation: keep the top-k, drop the rest

import numpy as np

M = np.random.randn(50, 30)
U, S, Vt = np.linalg.svd(M, full_matrices=False)

k = 5                                  # keep 5 directions
M_approx = U[:, :k] @ np.diag(S[:k]) @ Vt[:k]
energy = (S[:k]**2).sum() / (S**2).sum()
print(M_approx.shape)                  # (50, 30) but rank 5
print(f"kept {energy:.0%} of the variance with k=5")

Key Points to Remember

• 1SVD factors ANY matrix: M = U Σ Vᵀ (rotate → scale → rotate)
• 2Singular values rank directions by importance (energy/variance)
• 3Keeping the top-k singular values gives the best rank-k approximation
• 4Powers compression, denoising, latent-factor recommenders, and LoRA fine-tuning