Multiclass classification
Multiclass classification: predicting one label out of K (>2) classes, key methods (one-vs-rest, one-vs-one, softmax/multinomial logistic), evaluation (multiclass confusion matrix, per-class precision/recall, macro/micro averaging), and gradient derivation for softmax cross-entropy.
Drafted with Aria, reviewed by the AiCanCode.org team · Updated . How we write and check our notes. Spotted an error? Use Give Feedback at the bottom of the page.
Why it matters
Multiclass classification is used when an engineer must assign each input one label out of more than two possible labels — for example, handwritten digit recognition (0–9), traffic-sign recognition, or language identification. Correct model choice, training loss and evaluation metrics are essential to deploy reliable systems in computer vision, NLP and embedded ML.
Key ideas
Start from first principles and simple definitions; give exact statements used in exams.
Multiclass classification: the supervised learning problem where each input x must be assigned exactly one label y from a finite set of K distinct classes {1, 2, ..., K}. Assumption: classes are mutually exclusive and labels are single-label (not multilabel).
Feature vector x: an ordered list of measured quantities (x1, x2, ..., xd). Each xi is a feature. Unit depends on measurement; after standard scaling features are often unitless.
Hypothesis / model: a function h(x; θ) that maps input x to either (a) a score vector z ∈ R^K of unnormalised scores (called logits), or (b) a probability vector p ∈ [0,1]^K summing to 1.
Discriminant function: for linear models, K functions g_k(x) = w_k^T x + b_k produce scores; predicted label is argmax_k g_k(x).
One-vs-Rest (OvR): train K binary classifiers; classifier k separates class k (positive) from all others (negative). At test time, apply all K and pick the class whose classifier gives the highest confidence score. Assumptions and trade-offs: simple, works with any binary classifier, but class imbalance in each binary problem can cause poor calibration.
One-vs-One (OvO): train K*(K−1)/2 classifiers, one for every pair of classes (i,j). At test time, use voting or pairwise comparisons. More classifiers, but each trained on fewer examples; good when binary classifiers are cheap.
Softmax (multinomial logistic) regression: directly models class probabilities with the softmax function applied to logits z ∈ R^K: p_k = exp(z_k) / sum_{j=1..K} exp(z_j). Prediction: argmax_k p_k = argmax_k z_k. Assumption: single-label mutually exclusive classes. Softmax estimates class probabilities; calibration may require post-processing (e.g., temperature scaling).
Cross-entropy (negative log-likelihood) for a single example with one-hot label y (y_k = 1 for true class, else 0): L = − sum_{k=1..K} y_k log p_k. For one-hot y this reduces to L = − log p_{true}.
Exact derivative used in exams (stated as a theorem): For softmax followed by cross-entropy, the gradient of L with respect to the logits z is (p − y). Proof/derivation is given in Worked examples.
Multiclass confusion matrix: a K×K table where rows are true classes and columns are predicted classes. Entry C[i,j] = number of examples with true class i and predicted class j. From this we define per-class TP, FP, FN (see below) and compute precision/recall/F1 per class.
Per-class definitions (for class k):
- True Positive (TP_k) = C[k,k].
- False Positive (FP_k) = sum_{i≠k} C[i,k].
- False Negative (FN_k) = sum_{j≠k} C[k,j].
- True Negative (TN_k) = sum of all entries not in row k or column k.
Precision_k = TP_k / (TP_k + FP_k) (if denominator zero, define precision as 0 by convention in exam answers but mention this).
Recall_k (also sensitivity) = TP_k / (TP_k + FN_k).
F1_k = 2·Precision_k·Recall_k / (Precision_k + Recall_k) (harmonic mean).
Macro-average = arithmetic mean of the per-class metric across K classes (treats all classes equally).
Micro-average = compute global counts (sum TP, sum FP, sum FN) and then compute the metric (weights classes by support). For single-label multiclass, micro-precision = micro-recall = micro-F1 = overall accuracy.
Diagram to draw (for exams): for K=3 linear classifier in 2D, draw axes x1, x2; draw three linear decision boundaries as three straight lines; mark three regions with different colours labelled class 1,2,3. Also draw pairwise separating lines (between every pair of weight vectors) and label boundary equations w1^T x + b1 = w2^T x + b2 etc.
Assumptions to state in answers: whether classes are mutually exclusive, whether labels are balanced, whether features are scaled, and whether regularisation is used.
Formulas
p_k = exp(z_k) / sum_{j=1..K} exp(z_j) — p_k: predicted probability of class k, unitless; applies when modeling probabilities with softmax. For numerical stability, implement softmax with z' = z − max_j z_j before exponentiation (this does not change p).
z_k = w_k^T x + b_k — z_k: logit (score) for class k, unit: depends on x and w (usually unitless after scaling); applies for linear logits.
L = - sum_{k=1..K} y_k * log(p_k) — cross-entropy loss for one example; y_k is one-hot (1 for true class); dimensionless (in nats if natural log is used, in bits if log base 2); applies for softmax output.
dL/dz = p - y — gradient of cross-entropy loss with respect to logits z (K-vector), unitless.
dL/dW = x * (p - y)^T — gradient for weight matrix W (shape d×K) for a single example; if using minibatch average over examples, average gradients.
Accuracy = sum_{k=1..K} C[k,k] / N — N is total samples, unitless.
Precision_k = TP_k / (TP_k + FP_k) — per-class precision, unitless.
Recall_k = TP_k / (TP_k + FN_k) — per-class recall, unitless (if denominator is 0, define recall as 0 by the stated convention).
F1_k = 2 * Precision_k * Recall_k / (Precision_k + Recall_k) — per-class F1, unitless.
Macro_metric = (1/K) * sum_{k=1..K} metric_k — average across classes.
Weighted_metric = (sum_{k=1..K} support_k * metric_k) / (sum_{k=1..K} support_k) — support-weighted average, where support_k = TP_k + FN_k.
Micro_precision = (sum_k TP_k) / (sum_k TP_k + sum_k FP_k) — equals accuracy for single-label multiclass.
(Add L2 regulariser: L_reg = L + (λ/2) * ||W||^2; dL_reg/dW = dL/dW + λ W.)
Worked examples
Example 1 — Compute metrics from a 3-class confusion matrix (numerical, exam style). Data (rows=true, columns=predicted): Row A: [30, 5, 5] (total 40) Row B: [3, 25, 2] (total 30) Row C: [2, 3, 25] (total 30) Total N = 100. Step 1: Accuracy = (30 + 25 + 25) / 100 = 80 / 100 = 0.80.
Step 2: Predicted totals: Pred A = 30+3+2 = 35; Pred B = 5+25+3 = 33; Pred C = 5+2+25 = 32.
Per-class metrics:
- Class A: TP_A = 30, FP_A = 3+2 = 5 (sum of column A excluding row A), FN_A = 5+5 = 10. Precision_A = 30/35 = 0.857142857. Recall_A = 30/40 = 0.75. F1_A = 2*(0.857142857*0.75)/(0.857142857+0.75) = 0.80.
- Class B: TP=25, FP_B = 5+3 = 8, FN_B = 3+2 = 5. Precision_B = 25/33 ≈ 0.757575758. Recall_B = 25/30 ≈ 0.833333333. F1_B ≈ 0.793650794.
- Class C: TP=25, FP_C = 5+2 = 7, FN_C = 2+3 = 5. Precision_C = 25/32 = 0.78125. Recall_C = 25/30 ≈ 0.833333333. F1_C ≈ 0.80645161.
Macro-averages: Macro-Precision ≈ (0.857142857 + 0.757575758 + 0.78125)/3 ≈ 0.79866. Macro-Recall ≈ (0.75 + 0.833333333 + 0.833333333)/3 ≈ 0.80556. Macro-F1 ≈ (0.80 + 0.793650794 + 0.80645161)/3 ≈ 0.80003.
Micro-averages: sum TP = 80, total predicted = 100 ⇒ Micro-Precision = 80/100 = 0.80; Micro-Recall = 80/100 = 0.80; Micro-F1 = 0.80. Note: for single-label multiclass micro-F1 equals accuracy.
Example 2 — Derive gradient for softmax + cross-entropy (standard exam derivation). Given input x (d-vector), weight matrix W (d×K), bias b (K-vector). Logits z = W^T x + b (K-vector). Softmax: p_k = exp(z_k)/sum_j exp(z_j). Loss for one-hot y: L = − sum_k y_k log p_k.
Derivation steps:
Compute partial derivative of L wrt z_k. Using chain rule and softmax algebra (see proof below): dL/dz_k = p_k − y_k. (Proof sketch: for one example, d/dz_k (−log p_t) = −(1/p_t) * dp_t/dz_k. Then use dp_t/dz_k = p_t * (δ_{kt} − p_k). Substitute to obtain p_k − y_k.)
dL/dW: each column k of W receives gradient (dL/dz_k) * x. In matrix form: dL/dW = x * (p − y)^T (shape d×K).
For a minibatch average, average (p − y) across examples and multiply by X^T accordingly.
Regularisation: with L2 penalty (λ/2)||W||^2, add λ W to gradient.
Final boxed result (what exam answers must show): dL/dW = x (p − y)^T and dL/db = p − y, with dL/dz = p − y.
Common mistakes
- Using binary-specific metrics (e.g., ROC AUC without modifications) directly for multiclass without explaining OvR/OvO extension.
- Thinking micro-average is the same as macro-average; micro weights by class frequency, macro does not.
- Applying softmax when the problem is multilabel (non-exclusive labels); softmax enforces exclusivity.
- Forgetting to add regularisation gradient when deriving updates.
For your exam
Types of questions you'll see:
- Definitions and short notes: define multiclass classification, OvR vs OvO, softmax.
- Derivations: show gradient of softmax + cross-entropy; derive update rule including L2 regulariser.
- Numericals: compute accuracy, per-class precision/recall/F1 and macro/micro averages from a confusion matrix.
- Diagram questions: draw decision boundaries for linear 3-class classifier in 2D and label them.
What full-marks answers include:
- State assumptions (mutually exclusive classes, scaling, regularisation), show intermediate steps symbolically, label axes in diagrams, and compute numbers with units or state unitless. For derivations show chain rule steps and shapes of matrices.
Quick check
- For single-label multiclass with K classes, which averaging method treats all classes equally? 2) What is the gradient dL/dz for softmax + cross-entropy? 3) If sum of diagonals of confusion matrix is 150 and total samples are 200, what is accuracy? Answers: 1) Macro-average. 2) p − y. 3) Accuracy = 150/200 = 0.75.
Finished this topic? Mark it so your progress, study plan and readiness keep up.
Stuck on something here?