Vector Operations & the Dot Product

Overview

If you learn one operation in linear algebra, make it the dot product. You multiply two vectors element-by-element and add the results into one number. That number answers "how aligned are these two vectors?" — large and positive when they point the same way, zero when perpendicular, negative when opposed. This single idea powers an astonishing amount of AI: a neuron computes the dot product of its inputs with its weights; semantic search ranks documents by the dot product (cosine similarity) of their embeddings; the attention mechanism in transformers scores tokens by dot products of queries and keys. Once you see the dot product as a similarity/relevance score, attention and embeddings stop being mysterious.

Dot product = element-wise multiply, then sum

import numpy as np

a = np.array([1.0, 2.0, 3.0])
b = np.array([4.0, 5.0, 6.0])

# manual: 1*4 + 2*5 + 3*6 = 32
print(np.sum(a * b))    # 32.0
print(a @ b)            # 32.0  -> the @ operator is the dot product

As a similarity score: cosine similarity

import numpy as np

def cosine(a, b):
    return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))

query = np.array([1.0, 0.0, 1.0])
doc1  = np.array([1.0, 0.0, 0.9])   # similar direction
doc2  = np.array([0.0, 1.0, 0.0])   # perpendicular

print(round(cosine(query, doc1), 3))  # 0.986 -> very similar
print(round(cosine(query, doc2), 3))  # 0.0   -> unrelated

Key Points to Remember

• 1Dot product: multiply matching entries and sum into one number
• 2It measures alignment — large & positive = same direction, 0 = perpendicular
• 3A neuron is a dot product of inputs and weights (plus a bias)
• 4Cosine similarity = normalised dot product; the basis of embedding/RAG search