Artificial Intelligence & Data Science is coming soon

We're still writing and reviewing the notes for every subject, and we'll open it when every subject is ready. Meanwhile you can practise aptitude, ask Aria about any subject, or see the branches that are ready.

Models of machine learning

Overview of common families of machine-learning models, how they represent functions, their assumptions, and how to choose and evaluate them.

Drafted with Aria, reviewed by the AiCanCode.org team · Updated . How we write and check our notes. Spotted an error? Use Give Feedback at the bottom of the page.

Why it matters

Machine learning models are the mathematical structures engineers choose to map input data to predictions (labels, scores, reconstructions). Choosing the right model and understanding its assumptions controls accuracy, generalisation and computational cost in real systems (embedded devices, server backends, or offline analytics).

Key ideas

Definitions (bolded on first use):

  • Model: a set of functions or rules that map input features x to outputs y, chosen to approximate the unknown true mapping. A model can be described by a parametric family or a nonparametric rule.
  • Hypothesis space: the set H of all functions the model can represent. For example, for linear models H = {x ↦ w·x + b | w,b real}.
  • Parametric model: a model described by a fixed, finite-dimensional parameter vector θ ∈ R^d. Assumption: a correct or useful mapping is well approximated within this finite-dimensional family.
  • Nonparametric model: a model whose complexity grows with the data (e.g., k-NN, kernel density estimators). No fixed finite parameter vector is assumed.
  • Generative model: a model that defines a joint distribution p(x,y); we obtain p(y|x) via Bayes rule. Example: naive Bayes. Assumption: model of p(x|y) is accurate.
  • Discriminative model: a model that directly models p(y|x) or a decision rule y = f(x) without modelling p(x). Example: logistic regression, SVM. Assumption: focusing on decision boundary yields better predictive performance.
  • Likelihood: for observed data D and parameter θ, the function L(θ) = p(D|θ). Maximising L(θ) gives the maximum-likelihood estimate (MLE).
  • Prior and posterior (Bayesian): prior p(θ) expresses belief before data; posterior p(θ|D) ∝ p(D|θ)p(θ).

Basic theorems and facts (exact statements):

  • Maximum-likelihood estimator (MLE) definition: Given independent data x1,...,xn with joint density p(x1,...,xn|θ) = ∏{i=1}^n p(xi|θ), an MLE is any θ̂ satisfying θ̂ = argmax_θ ∏{i=1}^n p(xi|θ). Assumption: data are i.i.d. from the model family.
  • Bayes rule (exact): For events A and B with p(B)>0, p(A|B) = p(B|A)p(A)/p(B). In continuous form for random variables: p(θ|D) = p(D|θ)p(θ)/p(D), where p(D) = ∫ p(D|θ)p(θ) dθ.
  • Bias–variance trade-off (definition): For estimator f̂(x) of true function f(x), expected squared error decomposes as E[(f̂(x)-f(x))^2] = (Bias[f̂(x)])^2 + Var[f̂(x)] + irreducible noise. Assumption: squared loss and expectation over training sets.

Model families, their assumptions and when to use them:

  1. Linear and generalized linear models: assume the target is (approximately) a linear function of features or a linear function after transformation (link function). Easy, fast, interpretable. Draw: 2D scatter of points and a straight line; label axes x1 and y; for regression draw a fitted line y ≈ w·x + b; for classification in feature space draw the decision boundary w·x + b = 0 separating classes.

  2. Nearest neighbours (k-NN): nonparametric; classification by majority vote among k nearest training points measured by a distance (usually Euclidean). Assumptions: similar inputs have similar outputs (locality). Draw: 2D plane with labelled points and a query point; draw circle including k neighbours.

  3. Probabilistic generative models: e.g., naive Bayes assumes conditional independence of features given class. Assumption must be stated explicitly: p(x|y) = ∏_j p(x_j|y). Draw: two class-conditional Gaussian blobs and sketch class priors.

  4. Discriminative probabilistic models: logistic regression models p(y=1|x) = σ(w·x+b) where σ is the sigmoid. No modelling of p(x).

  5. Decision trees: partition feature space into axis-aligned (or oblique) regions with constant predictions. Assumption: data can be well-separated by recursive partitioning; prone to overfitting if not pruned. Draw: 2D feature plane partitioned into rectangular regions with class labels.

  6. Kernel and similarity methods (SVMs, kernel ridge): map inputs implicitly to high-dimensional feature spaces via a kernel K(x,x') (K should be symmetric positive semidefinite). Assumption: appropriate kernel exists that makes data (more) separable.

  7. Neural networks (MLP): parametric, universal function approximators (under mild assumptions: a network with one hidden layer and enough neurons with non-polynomial activation can approximate continuous functions on compact sets). Assumption: enough data and compute; risk of overfitting without regularisation.

Model selection principles:

  • Match model complexity to data size and noise: small data → prefer simpler parametric models or heavy regularisation. Large data → nonparametric or large neural nets can work.
  • Prefer models whose assumptions are approximately true (e.g., Gaussian class-conditional for LDA).
  • Use cross-validation to estimate generalisation error; prefer simpler model if performance is similar (Occam's razor).

Diagrams to draw in answers (describe to student):

  • For linear model fitting: draw scatter of points (x-axis input, y-axis output) and a fitted line. Label slope w and intercept b. Indicate residuals as vertical distances.
  • For k-NN: draw training points with two classes, a query point, and the circle of radius to the k-th neighbour showing which points count.
  • For generative vs discriminative: draw block diagram with p(x,y)→p(y|x) (generative path uses p(x|y) and p(y)).

Formulas

y = w·x + b — linear model. w: weight vector (unit depends on units of y per unit x), b: bias (same unit as y), applies for regression/linear prediction. p(y|x) = σ(w·x + b) where σ(t) = 1/(1+e^{-t}) — logistic regression probability for binary label y∈{0,1}. σ(t) unitless, applies for binary classification. p(θ|D) = p(D|θ) p(θ) / p(D) — Bayes rule (posterior). All terms are probability densities (unit: 1/(unit of θ) if continuous). Assumes p(D)>0. L(θ) = ∏_{i=1}^n p(x_i | θ) — likelihood for i.i.d. data. Unit: product of probability densities; use log-likelihood for optimisation. x_NN(x_query) = argmin_i d(x_query, x_i) — 1-NN: pick the training point closest to the query. For k-NN classification: ŷ(x_query) = argmax_c ∑_{i ∈ N_k(x_query)} 1[y_i = c]. For k-NN regression: ŷ(x_query) = (1/k) ∑_{i ∈ N_k(x_query)} y_i. Distances d are typically Euclidean; feature scaling affects d. µ̂ = (1/n) ∑_{i=1}^n x_i — MLE for mean of i.i.d. Gaussian with known variance. µ̂ units same as x, applies when data are i.i.d. Gaussian and variance known or unknown with standard MLE derivation. σ̂^2 = (1/n) ∑_{i=1}^n (x_i - µ̂)^2 — MLE for variance (note: MLE uses n in denominator; unbiased sample variance uses n-1). Units: (units of x)^2. p(y|x) = p(x|y) p(y) / ∑_{y'} p(x|y') p(y') — posterior for discrete classes. Probabilities (unitless). Assumes modelled class-conditionals p(x|y). d(x,x') = ||x - x'||_2 = sqrt(∑_{j} (x_j - x'_j)^2) — Euclidean distance, units same as features.

Worked examples

Example 1 — Naive Bayes (categorical features). Problem: Classify email as Spam (S) or Ham (H) given two binary features: F1 = contains word "offer" (Yes/No), F2 = contains word "meeting" (Yes/No). Training counts: 40 emails: 25 Spam, 15 Ham. Among Spam: 20 have offer, 3 have meeting. Among Ham: 1 has offer, 10 have meeting. Query email has offer=Yes, meeting=No. Classify with naive Bayes. Steps:

  1. Compute priors: p(S)=25/40=0.625, p(H)=15/40=0.375.
  2. Likelihoods (naive independence): p(F1=Yes|S)=20/25=0.8, p(F2=No|S)=1 - p(F2=Yes|S)=1 - 3/25 = 22/25 = 0.88. For Ham: p(F1=Yes|H)=1/15≈0.0667, p(F2=No|H)=1 - 10/15 = 5/15 = 0.3333.
  3. Compute unnormalised posteriors:
    • Score_S = p(S)·0.8·0.88 = 0.625 * 0.704 = 0.44.
    • Score_H = p(H)·0.0667·0.3333 ≈ 0.375 * 0.02222 ≈ 0.008333.
  4. Normalise if desired: posterior_S ≈ 0.44/(0.44+0.008333) ≈ 0.982. Posterior_H ≈ 0.018. Decision: predict Spam (posterior ≈ 98.2%).

Example 2 — k-NN in 2D (numerical distances). Problem: Training points: A(0,0), A(1,0), B(0,2), B(2,2). Query point q(1,1). Use k=3 and Euclidean distance. Steps:

  1. Distances: d(q,A(0,0)) = sqrt((1-0)^2 + (1-0)^2) = sqrt(2) ≈ 1.414. d(q,A(1,0)) = sqrt((1-1)^2 + (1-0)^2) = 1.0. d(q,B(0,2)) = sqrt((1-0)^2 + (1-2)^2) = sqrt(2) ≈ 1.414. d(q,B(2,2)) = sqrt((1-2)^2 + (1-2)^2) = sqrt(2) ≈ 1.414.
  2. Sort distances: 1.0 (A), 1.414 (A), 1.414 (B), 1.414 (B). If equal distances occur, assume ties are broken by the order shown. For k=3 nearest: labels = [A, A, B]. Majority = A (2 out of 3). Decision: predict A.

Example 3 — MLE for Gaussian mean and variance. Data (units: seconds): [2.0 s, 4.0 s, 3.0 s]. Assume i.i.d. Normal with unknown mean µ and unknown variance σ^2. MLEs:

  1. µ̂ = (1/3) (2 + 4 + 3) = 3.0 s. µ̂ = 3.0 s.
  2. σ̂^2 = (1/3)[(2-3)^2 + (4-3)^2 + (3-3)^2] = (1/3)(1 + 1 + 0) = 0.6666667 s^2. σ̂^2 = 0.6667 s^2.

Common mistakes

  • Confusing generative p(x,y) with discriminative p(y|x) and applying Bayes incorrectly.
  • Not scaling or normalising features before distance-based methods (k-NN, kernel methods), which changes distances across feature scales.
  • Interpreting posterior probabilities from non-probabilistic models (e.g., raw SVM distance is not a probability without calibration).

For your exam

Types of questions:

  • Definitions and short notes: define parametric vs nonparametric models, generative vs discriminative.
  • Derivations: derive MLE for Gaussian mean and variance; derive decision rule for Bayesian classifier.
  • Numericals: naive Bayes classification with counts; k-NN classification with given points.
  • Diagrams: draw decision boundaries for linear models vs trees; illustrate hypothesis space.
  • Comparison/discussion: choose an appropriate model for a described dataset, justify assumptions.

What a full-marks answer includes:

  • Precise definitions (bold key terms), stated assumptions (i.i.d., independence, noise model), and correct derivations with algebraic steps.
  • Where needed, clear diagrams labelled exactly (axes, decision boundary, points, residuals).
  • For numerical questions: show each computational step, units, intermediate values, and final answer highlighted.

Quick check

  1. Give the MLE µ̂ for data x = [1, 3, 6].
  2. For binary logistic regression, what function maps real values to probabilities?
  3. In naive Bayes, what independence assumption is made about features? Answers: 1) µ̂ = 10/3 ≈ 3.333. 2) The sigmoid σ(t)=1/(1+e^{-t}). 3) Features are conditionally independent given the class (p(x|y)=∏_j p(x_j|y)).

Finished this topic? Mark it so your progress, study plan and readiness keep up.

Stuck on something here?