Eigenvalues & Eigenvectors

Overview

Most vectors get knocked off their direction when you apply a matrix, but a few special ones — the eigenvectors — keep pointing the same way and merely get scaled by a factor, the eigenvalue. Formally, Av = λv. These directions are the "natural axes" of a transformation. Their headline use in AI is Principal Component Analysis (PCA): the eigenvectors of a dataset's covariance matrix are the directions of greatest variance, so keeping the top few lets you compress high-dimensional data (dimensionality reduction) while preserving most of its information. Eigenvalues also tell you about stability (do repeated applications blow up or shrink?), which matters for understanding exploding/vanishing gradients in deep and recurrent networks.

Av = λv: directions that only get scaled

import numpy as np

A = np.array([[2.0, 0.0],
              [0.0, 3.0]])
vals, vecs = np.linalg.eig(A)
print(vals)                 # [2. 3.]  -> eigenvalues
print(vecs)                 # columns are eigenvectors (here the x and y axes)

v = vecs[:, 1]              # eigenvector for lambda = 3
print(np.allclose(A @ v, 3.0 * v))   # True -> Av = λv

PCA: eigenvectors of the covariance = axes of variance

import numpy as np

X = np.random.randn(200, 5)          # 200 samples, 5 features
X = X - X.mean(axis=0)               # center the data
cov = np.cov(X, rowvar=False)        # 5x5 covariance
vals, vecs = np.linalg.eigh(cov)     # eigh: symmetric matrices

top2 = vecs[:, np.argsort(vals)[::-1][:2]]   # 2 largest-variance directions
X_2d = X @ top2                      # compress 5-D -> 2-D
print(X_2d.shape)                    # (200, 2)

Key Points to Remember

• 1Eigenvector: a direction a matrix scales but does not rotate (Av = λv)
• 2Eigenvalue λ: the scaling factor along that direction
• 3PCA uses the top eigenvectors of the covariance for dimensionality reduction
• 4Eigenvalues indicate stability — relevant to exploding/vanishing gradients