Normalisation and distance measures
Normalisation rescales attributes to a common range; distance measures (Euclidean, Manhattan, Minkowski, cosine, Hamming, etc.) quantify similarity or dissimilarity between records — both are essential before distance-based data mining (clustering, k-NN) to avoid bias from scale or type differences.
Drafted with Aria, reviewed by the AiCanCode.org team · Updated . How we write and check our notes. Spotted an error? Use Give Feedback at the bottom of the page.
Why it matters
Normalisation and distance measures are used every time you compare records or records-to-centres: in clustering, k‑nearest neighbours, similarity searches and many parts of the KDD process. If attributes have different units or ranges, raw distances are biased; choosing the right distance function and normalisation fixes that and improves model quality.
Key ideas
Start with basic words: an attribute is a variable/column in the dataset. A record (or instance, vector) is one data point with values for attributes.
Normalisation (scaling): any transformation that rescales an attribute's values so different attributes become comparable. It does not create information, but changes shape of the space used by distance functions.
Distance (dissimilarity) measure: a function D(x,y) giving how different two records x and y are. If D satisfies the four metric axioms it is a metric:
- Non-negativity: D(x,y) ≥ 0 for all x,y.
- Identity of indiscernibles: D(x,y) = 0 iff x = y.
- Symmetry: D(x,y) = D(y,x).
- Triangle inequality: D(x,z) ≤ D(x,y) + D(y,z). Note: sometimes a similarity (higher = more similar) is used; similarity and distance are different types of measures.
Minkowski distance (general definition): for two k-dimensional numeric vectors x and y and parameter p ≥ 1, the Minkowski distance is defined by the p‑norm. Special cases:
- p = 1: Manhattan distance (L1) — sum of absolute differences.
- p = 2: Euclidean distance (L2) — straight-line distance in Euclidean space.
- p → ∞: Chebyshev distance — maximum absolute difference across coordinates.
Squared Euclidean distance: same as Euclidean but without the square root. It preserves ordering of distances and is often used to save computation (e.g., k‑means objective).
Cosine similarity: measures the cosine of the angle between two vectors. It depends only on direction, not magnitude. From cosine similarity you can make a distance: cosine distance = 1 − cosine_similarity, or use angular distance = arccos(cosine_similarity) which is a metric.
Hamming distance: for categorical or binary attributes, count of positions with different values.
Mahalanobis distance: D_M(x,y) = sqrt((x − y)^T S^{-1} (x − y)), where S is the covariance matrix of the features; accounts for scale and correlation, is unitless, and requires S to be positive definite (invertible).
Gower distance: a method to combine numeric and categorical attributes into a single dissimilarity (useful for mixed data). (I state its existence; the full formula is longer — verify if you want the detailed coefficient forms.) For observations i and j with p attributes and weights w_k (handling missing values), define Gower similarity S_ij = (Σ_k w_k s_ijk)/(Σ_k w_k), where for numeric attributes s_ijk = 1 − |x_ik − x_jk|/range_k and for categorical attributes s_ijk = 1 if values match else 0 (asymmetric binary can be handled via w_k). The Gower distance is D_ij = 1 − S_ij.
Common normalisation methods:
- Min‑max normalisation: rescales to [0,1] (or [a,b]). Requires known min and max. Be careful: new data outside the original range can fall outside [0,1].
- Z‑score standardisation (standardisation): rescale to zero mean and unit variance using mean and standard deviation. Decide if you use population SD (divide by N) or sample SD (divide by N−1) — be consistent.
- Robust scaling (median–IQR): x' = (x − median)/IQR, less sensitive to outliers; IQR = Q3 − Q1.
- Decimal scaling: divide by 10^j so that values lie in [−1,1] (typically < 1, except when the maximum absolute value is an exact power of 10).
- Unit vector normalisation (L2 normalisation): scale a record so its L2 norm equals 1. This is common when using cosine similarity.
Derivations universities ask for (one example): show relation between cosine and Euclidean for unit vectors. Derivation: if ||x|| = ||y|| = 1 then ||x − y||^2 = (x − y)·(x − y) = ||x||^2 + ||y||^2 − 2 x·y = 1 + 1 − 2 cosθ = 2(1 − cosθ). So squared Euclidean distance between unit vectors equals 2(1 − cosine_similarity). Draw points x and y on the unit circle, label angle θ between them and show triangle with chords; compute dot product and lengths.
When an exam wants a diagram: draw a 2D Cartesian plane, label axes X and Y, plot points A(x1,y1) and B(x2,y2), draw vector from origin to A and B, draw the segment AB (label its length). Draw right triangle by projecting B−A on axes; label horizontal difference Δx = x2−x1 and vertical Δy = y2−y1; compute Euclidean distance as sqrt(Δx^2 + Δy^2).
Formulas
minmax(x) = (x - min_X) / (max_X - min_X) — x value, min_X and max_X are the minimum and maximum of that attribute; unitless; applies when max_X ≠ min_X.
minmax_to_ab(x) = a + (x - min_X) * (b - a) / (max_X - min_X) — scales to [a,b]; a,b are target bounds; unitless.
z = (x - μ) / σ — z score; μ is mean of attribute (same units as x), σ is standard deviation (same units); result is unitless. Use population σ = sqrt(Σ(x−μ)^2 / N) or sample σ = sqrt(Σ(x−μ)^2 / (N−1)).
robust_z = (x − median_X) / IQR_X — median and interquartile range; unitless; robust to outliers.
decimal_scaling(x) = x / 10^j where j = ceil(log10(max(|x_i|))) — result unitless.
unit_vec(x) = x / ||x||_2 where ||x||_2 = sqrt(Σ_i x_i^2) — unitless.
Minkowski_p(x,y) = (Σ_i |x_i - y_i|^p)^(1/p) — p ≥ 1; unit = same as underlying attributes (if attributes have units, the distance carries those units); applies to numeric vectors.
Euclidean(x,y) = sqrt(Σ_i (x_i - y_i)^2) — special case p=2; unit: same as attributes.
Squared_Euclidean(x,y) = Σ_i (x_i - y_i)^2 — avoids sqrt; unit squared.
Manhattan(x,y) = Σ_i |x_i - y_i| — p=1.
Chebyshev(x,y) = max_i |x_i - y_i| — p→∞.
Mahalanobis(x,y) = sqrt((x − y)^T S^{-1} (x − y)) — S is the covariance matrix; unitless; defined when S is positive definite.
Cosine_similarity(x,y) = (x·y) / (||x||·||y||) — unitless; requires nonzero vectors.
Cosine_distance(x,y) = 1 - Cosine_similarity(x,y) — unitless; note: this is not always a metric; angular distance arccos(Cosine_similarity) is.
Hamming(x,y) = count(i : x_i ≠ y_i) — for categorical/binary attributes; unitless.
Worked examples
Example 1 — Normalisation (numeric attribute: Height in cm). Data: [150, 160, 170, 180] cm.
- Min‑max to [0,1]. min = 150, max = 180, range = 30.
minmax(150) = (150-150)/30 = 0.000(0.000)minmax(160) = (160-150)/30 = 10/30 = 0.333...(0.333)minmax(170) = 20/30 = 0.666...(0.667)minmax(180) = 30/30 = 1.000(1.000) - Z‑score (use population SD here). Mean μ = (150+160+170+180)/4 = 165 cm.
Variance = [225 + 25 + 25 + 225]/4 = 125 (cm^2). σ = sqrt(125) = 11.18034 cm.
z(150) = (150-165)/11.18034 = -1.34164(-1.3416)z(160) = -0.44721(-0.4472)z(170) = 0.44721(0.4472)z(180) = 1.34164(1.3416) - Unit vector normalisation (L2): vector norm = sqrt(150^2 + 160^2 + 170^2 + 180^2) = sqrt(109400) = 330.7576. Normalised values: 150/330.7576 = 0.454, 160/330.7576 = 0.484, 170 -> 0.514, 180 -> 0.544 (rounded). Final 0.454, 0.484, 0.514, 0.544.
Example 2 — Distances between 3D points. Let A = (1,2,3), B = (4,0,-1).
Δ = B − A = (3, −2, −4).
Euclidean: sqrt(3^2 + (-2)^2 + (-4)^2) = sqrt(9+4+16) = sqrt(29) = 5.3851648 (5.3852)
Squared Euclidean: 29 (29)
Manhattan: |3| + |−2| + |−4| = 3 + 2 + 4 = 9 (9)
Minkowski with p=3: (27 + 8 + 64)^(1/3) = 99^(1/3) = 4.626... (4.626)
Cosine similarity: dot = 14 + 20 + 3*(−1) = 1. Norms: ||A|| = sqrt(14) = 3.741657, ||B|| = sqrt(17) = 4.123106. cos = 1/(3.741657*4.123106) = 1/√238 ≈ 0.06482. Cosine distance = 1 − 0.06482 = 0.93518 (0.9352)
Note: squared Euclidean is often used inside k‑means objective; cosine similarity is used for text and high‑dimensional data where direction matters.
Common mistakes
- Applying Euclidean distance to categorical attributes without encoding or using Hamming/Gower.
- Forgetting to handle zero variance (max = min or σ = 0) — you must detect and treat that attribute as constant (drop or keep but it contributes nothing to distance).
- Confusing cosine similarity with Euclidean distance; cosine ignores magnitude unless vectors are normalized.
- Using squared Euclidean in places where metric properties (triangle inequality) are required without justification.
For your exam
Types of questions:
- Definitions and short notes: define normalisation, list and compare methods, define metric axioms.
- Derivations: show relation between cosine and Euclidean for unit vectors, show Minkowski ⇒ Euclidean for p=2.
- Numericals: compute min‑max, z‑score, unit vector normalisations; compute distances for small vectors.
- Programs/traces: step-by-step k‑NN distance calculations after normalisation.
- Diagrams: draw 2D points and label Δx, Δy and compute Euclidean distance.
What full‑marks answers include: clear definitions with bold names, formulae with symbols explained, stepwise computation with units where relevant, and comments about assumptions (population vs sample SD, training vs test normalisation).
Quick check
- If x = 45, min = 30, max = 60, what is min‑max(x) to [0,1]? 2) For vector x=(3,4), what is its unit vector (L2 normalised) coordinates? 3) Two 2D unit vectors have cosine similarity 0.5; what is their squared Euclidean distance? Answers: 1) (45-30)/(60-30) = 15/30 = 0.5. 2) ||x|| = 5 so unit = (0.6, 0.8). 3) squared distance = 2(1 - 0.5) = 1.0
Finished this topic? Mark it so your progress, study plan and readiness keep up.
Stuck on something here?