Transpose, Identity & Inverse Matrices

Overview

Three utility operations show up constantly. The transpose swaps rows and columns; you use it to line up shapes for a matrix multiply (attention computes Q·Kᵀ, a transpose) and to move between "features in rows" and "features in columns". The identity matrix I has 1s on its diagonal and 0s elsewhere and behaves like the number 1: AI = A. The inverse A⁻¹ undoes a matrix: A·A⁻¹ = I, the way dividing undoes multiplying — it is how you "solve" a linear system in closed form (e.g., the normal equations for linear regression). Not every matrix has an inverse (singular matrices do not), which is itself an important signal about redundant or collinear data. In deep learning we rarely invert giant matrices (too expensive/unstable) and prefer gradient descent, but the concepts anchor your understanding of what layers do.

Transpose: swap rows and columns to fix shapes

import numpy as np

A = np.array([[1, 2, 3],
              [4, 5, 6]])      # (2, 3)
print(A.T.shape)              # (3, 2)

# Attention scores need queries · keys^T:
Q = np.random.randn(4, 8)     # 4 tokens, 8-dim
K = np.random.randn(4, 8)
scores = Q @ K.T              # (4, 8) @ (8, 4) -> (4, 4)
print(scores.shape)           # (4, 4): every token scored against every token

Identity and inverse: the "1" and the "undo"

import numpy as np

I = np.eye(3)                       # identity 3x3
A = np.array([[2.0, 1.0],
              [1.0, 3.0]])
A_inv = np.linalg.inv(A)

print(np.allclose(A @ A_inv, np.eye(2)))   # True -> A · A⁻¹ = I

# Solve Ax = b (e.g., closed-form regression) — prefer solve() over inv():
b = np.array([5.0, 10.0])
x = np.linalg.solve(A, b)
print(np.round(x, 3))               # the exact solution

Key Points to Remember

• 1Transpose (.T) swaps rows/columns — used to align shapes and compute Q·Kᵀ
• 2Identity matrix I is the "1": AI = IA = A
• 3Inverse A⁻¹ undoes A (A·A⁻¹ = I) and solves Ax = b in closed form
• 4Singular (non-invertible) matrices signal redundant/collinear data; prefer solve() over inv()