Scalars, Vectors, Matrices & Tensors

Overview

All data in AI — a price, a word embedding, a grayscale image, a batch of RGB images — is stored as numbers in a grid. A scalar is a single number (0-D). A vector is a list of numbers (1-D) and is how we represent one example: a data point, or a word's embedding. A matrix is a 2-D grid (rows × columns) and typically holds a batch of examples or the weights of a layer. A tensor is the general term for any number of dimensions; a batch of colour images is a 4-D tensor (batch, height, width, channels). PyTorch and TensorFlow are literally named after this object because everything a model does is tensor in, tensor out. Getting fluent with shapes — reading (32, 784) as "32 examples, 784 features each" — prevents the single most common bug in ML code: a shape mismatch.

The four containers and their shapes

import numpy as np

scalar = np.array(7.0)                       # 0-D
vector = np.array([1.0, 2.0, 3.0])           # 1-D: one example / embedding
matrix = np.array([[1, 2, 3], [4, 5, 6]])    # 2-D: 2 examples x 3 features
tensor = np.zeros((32, 28, 28, 3))           # 4-D: batch of 32 RGB images

for a in (scalar, vector, matrix, tensor):
    print(a.ndim, a.shape)
# 0 ()            1 (3,)        2 (2, 3)      4 (32, 28, 28, 3)

Shapes tell the story of your data

import numpy as np

# 4 sentences, each 10 tokens, each token a 512-dim embedding:
x = np.random.randn(4, 10, 512)
print(x.shape)          # (4, 10, 512)  -> (batch, seq_len, embed_dim)

# Grab the embedding of the 1st token of the 1st sentence:
print(x[0, 0].shape)    # (512,)  -> a single vector

Key Points to Remember

• 1Scalar (0-D), vector (1-D), matrix (2-D), tensor (N-D) — numbers in a grid
• 2A vector = one example or embedding; a matrix = a batch or a weight layer
• 3.shape reads left-to-right as the meaning of each axis (batch, features, …)
• 4Most ML bugs are shape mismatches — print shapes constantly