Computer Engineering is coming soon

We're still writing and reviewing the notes for every subject, and we'll open it when every subject is ready. Meanwhile you can practise aptitude, ask Aria about any subject, or see the branches that are ready.

Classification basics and imbalanced data

Basics of binary classification, the Bayes optimal rule, confusion-matrix metrics, problems caused by imbalanced classes, and practical fixes (sampling, class weights, thresholding, cost-sensitive learning, SMOTE) with exam-style derivations and worked examples.

Drafted with Aria, reviewed by the AiCanCode.org team · Updated . How we write and check our notes. Spotted an error? Use Give Feedback at the bottom of the page.

Why it matters

Many real engineering problems ask a model to decide between categories: spam vs not-spam, defective vs non-defective, churn vs stay. Classification with imbalanced data is common (rare faults, fraud, disease). Engineers must know how the optimal decision rule is formed, how to measure performance when classes are imbalanced, and which remedies (sampling, class weights, thresholding) to apply.

Key ideas

Start from first principles and build up.

  • Random variable: a quantity that can take values according to a probability distribution. Here x (feature vector) and y (class label) are random variables.
  • Classification problem: given an input x, predict label y ∈ {0,1} (binary case) or {1,...,K} (multiclass). We study binary for clarity; multiclass follows by extension.
  • Prior probability: P(y=c) is the probability that a random example belongs to class c before observing x. Priors reflect class imbalance when P(y=1) ≪ P(y=0).
  • Class-conditional density: p(x|y=c) is the probability density (or mass) of observing x given class c.
  • Posterior probability: by Bayes' theorem, P(y=c|x) = p(x|y=c) P(y=c) / p(x). This gives probability of class after seeing x.

Exact statement (Bayes classifier):

  • Let L(y, ŷ) be 0–1 loss (0 if correct, 1 if wrong). The classifier that minimises expected 0–1 loss for a given x is:
    • Bayes classifier: choose ŷ = argmax_c P(y=c | x). This rule assumes known priors and class-conditional densities and minimises the probability of error (the Bayes error).

Derivation of Bayes decision rule (short):

  1. Expected 0–1 loss given x for predicting class a is sum_{b} L(b,a) P(y=b|x). For 0–1 loss this equals 1 - P(y=a|x).
  2. Minimising 1 - P(y=a|x) over a is same as maximising P(y=a|x). Hence pick class with largest posterior.

Decision with unequal costs (cost-sensitive):

  • If cost C(a,b) is cost of predicting a when true label is b, choose a that minimises expected cost ∑_b C(a,b) P(y=b|x). For binary classes with cost of false negative C_fn and false positive C_fp, the decision boundary shifts: choose ŷ=1 when P(y=1|x) / P(y=0|x) > (C_fp - C_tn)/(C_fn - C_tp) Equivalently, choose ŷ=1 when P(y=1|x) > τ where τ = (C_fp - C_tn)/((C_fn - C_tp) + (C_fp - C_tn)). (For 0–1 loss C_tp=C_tn=0 this reduces to P(y=1|x)>P(y=0|x)).

Graphical: For one-dimensional x and two class-conditional densities, draw two curves p(x|y=0) and p(x|y=1), label x-axis, vertical axis density; mark the point x* where p(x|1)P1 = p(x|0)P0 (posterior equal). This is the decision threshold.

Confusion-matrix based metrics:

  • Define the 2×2 confusion matrix entries: TP (true positives) = count of actual 1 predicted 1; FP = actual 0 predicted 1; FN = actual 1 predicted 0; TN = actual 0 predicted 0. Always draw a 2×2 table with rows = actual class (top row actual=1, bottom actual=0) and columns = predicted (left predicted=1, right predicted=0), fill entries.

Important evaluation measures (why imbalance matters): overall accuracy = (TP+TN)/Total can be misleading when one class dominates. Prefer measures that focus on the minority class:

  • Precision = TP / (TP + FP): fraction of predicted positives that are correct.
  • Recall (Sensitivity, TPR) = TP / (TP + FN): fraction of actual positives found.
  • Specificity (TNR) = TN / (TN + FP): fraction of negatives correctly found.
  • F1 score = harmonic mean of precision and recall = 2·(Precision·Recall)/(Precision+Recall).
  • Balanced accuracy = (Recall + Specificity)/2.
  • Matthews correlation coefficient (MCC): a balanced measure even for imbalance; exact formula in Formulas section. For multiclass or skewed data, report macro-averaged precision/recall/F1 (unweighted mean across classes) and micro-averaged (global over all decisions); macro averaging better reflects minority-class performance.

ROC and PR curves:

  • ROC curve plots TPR vs FPR (FPR = FP/(FP+TN)) as decision threshold varies. AUC-ROC measures area under ROC.
  • When classes are highly imbalanced, Precision–Recall (PR) curve is more informative; AUC-PR focuses on performance on positive class. The no-skill baseline precision equals the positive class prevalence P(y=1); the baseline AUC-PR also equals P(y=1).

Remedies for imbalanced data (practical):

  • Resampling: undersampling removes majority examples; oversampling duplicates minority examples. Risk: undersampling may lose information; naive oversampling may overfit.
  • SMOTE (Synthetic Minority Over-sampling Technique): create synthetic minority examples by linear interpolation between a minority sample and its k nearest minority neighbors. Steps: for a minority sample x_i, pick neighbor x_j, create x_new = x_i + δ·(x_j - x_i) with random δ∈(0,1). Works in feature space; scale/normalise features first; use SMOTE variants (e.g., SMOTE-NC) for categorical or mixed data.
  • Class weights / cost-sensitive learning: give higher misclassification cost to minority class during training. In many classifiers (logistic regression, SVM, tree-based) you can pass class_weight or sample_weight.
  • Threshold moving: for probabilistic classifiers, change prediction threshold away from 0.5 to account for priors or costs.
  • Ensemble methods: boosting or balanced bagging that focus on minority errors.
  • Stratified splits/CV: use stratified train/validation/test splits and stratified k-fold cross-validation to preserve class ratios and obtain reliable estimates.

Assumptions to state: Bayes rule derivations assume known p(x|y) and P(y). Gaussian example below assumes class-conditionals are normal with known means and variances.

Formulas

P(y=c|x) = p(x|y=c)·P(y=c) / p(x) — P(y=c|x) is posterior, unitless, applies always when densities known. p(x) = sum_c p(x|y=c)·P(y=c) — marginal density of x, unit same as p(x|...), applies always. Decision (0–1 loss): ŷ = argmax_c P(y=c|x) — choose class with highest posterior. Accuracy = (TP + TN) / (TP + TN + FP + FN) — unitless, applies to classifier outcome counts. Precision = TP / (TP + FP) — unitless, requires (TP+FP)>0. Recall = TP / (TP + FN) — unitless, requires (TP+FN)>0. Specificity = TN / (TN + FP) — unitless, requires (TN+FP)>0. F1 = 2·Precision·Recall / (Precision + Recall) — unitless, requires Precision+Recall>0. BalancedAccuracy = (Recall + Specificity) / 2 — unitless. FPR = FP / (FP + TN) — unitless, requires (FP+TN)>0. MCC = (TP·TN - FP·FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)) — unitless, ranges in [-1, 1] when denominator>0. Log-likelihood ratio decision: choose y=1 if ln(p(x|1)/p(x|0)) + ln(P1/P0) > 0 — derived from Bayes, applies when densities positive.

If class-conditionals are Gaussians with common variance σ^2, and means μ0, μ1, decision reduces to linear threshold: For σ^2 common: choose y=1 if (x·(μ1-μ0) - (μ1^2 - μ0^2)/2)/σ^2 + ln(P1/P0) > 0 — unit: x and μ have same units; applies under Gaussian common-variance assumption.

Worked examples

Example 1 — Confusion-matrix metrics with imbalance (numerical, exam style): Data: 1000 test samples. Actual positives = 50 (rare class), actual negatives = 950. A classifier predicts 200 positives; among those predicted positives 40 are correct.

  1. Compute TP, FP, FN, TN:
    • TP = 40 (given)
    • FP = predicted_pos - TP = 200 - 40 = 160
    • FN = actual_pos - TP = 50 - 40 = 10
    • TN = actual_neg - FP = 950 - 160 = 790
  2. Accuracy = (TP + TN)/1000 = (40 + 790)/1000 = 830/1000 = 0.83
  3. Precision = TP/(TP+FP) = 40/200 = 0.20
  4. Recall = TP/(TP+FN) = 40/50 = 0.80
  5. F1 = 2·(0.20·0.80)/(0.20+0.80) = 0.32 => 0.32
  6. Specificity = TN/(TN+FP) = 790/950 ≈ 0.832
  7. Balanced accuracy = (0.80 + 0.832)/2 ≈ 0.816
  8. MCC = (TP·TN - FP·FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN))
    • Numerator = 40·790 - 160·10 = 31600 - 1600 = 30000
    • Denominator = sqrt(200·50·950·800) ≈ sqrt(7.6e9) ≈ 87177.98
    • MCC ≈ 30000 / 87177.98 ≈ 0.344 Comment: High accuracy (0.83) hides poor precision (0.20). In imbalanced settings precision/recall, F1 or MCC give more honest view.

Example 2 — Bayes decision boundary for Gaussian class-conditionals with imbalance (derivation + numeric): Assume x ∈ R, class-conditional densities p(x|0) = N(x; μ0, σ^2), p(x|1) = N(x; μ1, σ^2) with σ^2 known and equal. Let μ0 = 0, μ1 = 2, σ^2 = 1, priors P(y=1)=0.1, P(y=0)=0.9. Derive decision rule:

  1. Compare ln(p(x|1)P1) vs ln(p(x|0)P0). With equal variance, ln p(x|c) = - (x-μ_c)^2/(2σ^2) + const. Cancel constants.
  2. Inequality: - (x-μ1)^2/(2σ^2) + ln P1 > - (x-μ0)^2/(2σ^2) + ln P0 => ((x-μ0)^2 - (x-μ1)^2)/(2σ^2) + ln(P1/P0) > 0
  3. For numbers: ((x-0)^2 - (x-2)^2)/2 + ln(0.1/0.9) > 0 Expand: (x^2 - (x^2 -4x +4))/2 + ln(1/9) = (4x - 4)/2 - ln 9 = 2x - 2 - 2.197224577... So decision: 2x - 2 - 2.197224577 > 0 => 2x > 4.197224577 => x > 2.098612289.
  4. So classifier predicts y=1 only when x > 2.0986 (threshold shifted right because P(y=1) is small).
  5. Bayes error (probability of misclassification): P(error) = P1·P(x ≤ 2.0986 | y=1) + P0·P(x > 2.0986 | y=0).
    • For y=1: z1 = (2.0986 - 2)/1 = 0.0986 => Φ(z1) ≈ 0.5393 so P(x ≤ threshold |1) ≈ 0.5393
    • For y=0: z0 = (2.0986 - 0)/1 = 2.0986 => 1 - Φ(z0) ≈ 0.0179
    • Error ≈ 0.1·0.5393 + 0.9·0.0179 = 0.05393 + 0.01611 ≈ 0.0700 (≈7.00%) Note: check the normal CDF values if strict exam marking requires more precision.

Common mistakes

  • Forgetting to state or adjust decision threshold (0.5 is not always optimal under imbalance or non-equal costs).
  • Miscomputing MCC denominator terms or not checking for zero division when some sums are zero.
  • Applying SMOTE on data without normalising features first (interpolation in mixed-scale space produces bad samples).
  • Using AUC-ROC only when positive class is rare — PR AUC is more informative in that case.

For your exam

Question types you will see:

  • Definitions: define Bayes classifier, posterior, precision, recall, F1, specificity, MCC.
  • Short notes: explain why accuracy is misleading for imbalanced data; list remedies (with pros/cons).
  • Derivations: derive the Bayes decision rule from expected 0–1 loss; derive threshold for Gaussian class-conditionals (show algebra steps as in Worked Example 2).
  • Numericals: confusion matrix and compute all metrics (carry units as unitless probabilities); compute Bayes error given distributions and priors.
  • Diagrams: draw two class-conditional density curves on same axis and mark the decision threshold; label axes and densities, show region of error for each class. Full-marks answers include: precise definitions (bold terms), assumptions stated, step-by-step algebra for derivations, clearly labelled confusion matrix for numericals, and final numeric answers with correct significant figures.

Quick check

  1. What is the Bayes classifier decision rule under 0–1 loss?
  2. Give two reasons why accuracy can be misleading on imbalanced data.
  3. In a dataset with TP=30, FP=70, FN=20, TN=880, compute precision and recall. Answers: 1) Choose class with highest posterior P(y|x). 2) (a) A majority-class predictor can have high accuracy despite missing minority; (b) accuracy mixes effects on both classes and hides minority performance—use precision/recall/F1. 3) Precision = 30/(30+70)=0.30; Recall = 30/(30+20)=0.60.

Finished this topic? Mark it so your progress, study plan and readiness keep up.

Stuck on something here?