Bayes and naive Bayes classification
Bayes theorem gives a way to update class probabilities from evidence; naive Bayes uses a conditional-independence assumption to make classification simple, fast and effective for many engineering problems like spam detection and fault diagnosis.
Drafted with Aria, reviewed by the AiCanCode.org team · Updated . How we write and check our notes. Spotted an error? Use Give Feedback at the bottom of the page.
Why it matters
Bayes and naive Bayes classification are used by engineers to convert observed features into probabilities for classes (for example: spam or not-spam, defective or good). The method is simple, needs little training data, and is a standard baseline in machine-learning tasks and in larger systems such as text classification, medical diagnosis and feature-selection pipelines.
Key ideas
Bayes' theorem (defined exactly): For two events A and B with P(B) > 0, P(A | B) = P(B | A) · P(A) / P(B). Assumptions: probabilities are well-defined; P(B) ≠ 0. Derivation: from the definition P(A ∩ B) = P(A | B)P(B) = P(B | A)P(A), rearrange. Diagram to draw for an exam: draw a rectangle for the sample space, divide into two vertical regions labelled "C" and "not C" with widths proportional to P(C) and P(not C). From each region draw arrows to an event X with labels P(X | C), P(X | not C). Label the joint areas P(C)·P(X | C) and P(not C)·P(X | not C). Then show P(C | X) as the joint P(C ∩ X) divided by total P(X).
Posterior probability (defined): P(C | X), the probability of class C after observing features X (evidence X).
Prior probability (defined): P(C), the probability of class C before seeing X (often estimated from class frequencies).
Likelihood (defined): P(X | C), probability (or density) of observing X if the true class is C.
Naive Bayes classifier (defined): a classifier that applies Bayes' theorem under the conditional independence assumption (defined) that features x1, x2, ..., xd are independent given the class. Under this assumption: P(C | x1, x2, ..., xd) ∝ P(C) · Π_{i=1..d} P(xi | C). Assumptions: features are conditionally independent given class (this is often false in real data but the classifier still works well in many domains).
Decision rule (MAP) (defined): choose the class C that maximizes the posterior probability (maximum a posteriori): argmax_C P(C | X). Because denominator P(X) is same for all classes, equivalently choose argmax_C P(C) · Π P(xi | C).
Handling different feature types:
- For binary features, model P(xi = 1 | C) with Bernoulli counts.
- For multinomial / count features (text), use the multinomial model (counts per token) and estimate token probabilities per class. The class-conditional likelihood for a document x with word counts c_v is proportional to ∏_{v ∈ V} P(v | C)^{c_v}.
- For continuous features, common choice is the Gaussian (normal) distribution: model P(xi | C) as N(μ_{i,C}, σ_{i,C}^2) and estimate mean and variance from class data.
Parameter estimation: typically use maximum likelihood estimation (MLE): priors from class frequencies, for Gaussian mean use sample mean (sum/n) and for variance either the MLE (sum squared deviations / n) or unbiased sample variance (sum squared deviations / (n-1)) depending on convention. For discrete models use counts.
Smoothing (Laplace smoothing) (defined): to avoid zero probability when a feature value never appears in training for a class, add a small constant (commonly 1) to counts. This changes P(x | C) estimate to (count + α)/(n_C + α·k) where k is number of possible feature values; for multinomial text use the total token count T_C in class C as the denominator term instead of n_C and set k = |V| (vocabulary size).
Numerical stability: computing products of many small likelihoods causes underflow. Use the log form: log P(C | X) ∝ log P(C) + Σ log P(xi | C). This turns products into sums.
Formulas
P(A | B) = P(B | A) · P(A) / P(B) — P(A | B) is conditional probability of A given B (unitless); applies when P(B) > 0.
P(C | x1,...,xd) ∝ P(C) · ∏_{i=1..d} P(xi | C) — P(C | ...) is posterior (unitless); applies under the conditional independence assumption; proportional because divide by P(x1,...,xd) to get equality.
Class = argmax_C [ log P(C) + Σ_{i=1..d} log P(xi | C) ] — choose class maximizing log-posterior; logs are natural log (unitless); use when implementing to avoid numerical underflow.
Gaussian density: P(x | C) = (1 / sqrt(2·π·σ_C^2)) · exp( - (x - μ_C)^2 / (2·σ_C^2) ) — μ_C is mean of feature x in class C (same units as x), σ_C^2 is variance (units of x squared), applies when modeling continuous feature as Gaussian.
MLE mean: μ_C = (1 / n_C) · Σ_{j in class C} x_j — μ_C same units as x; n_C is number of training examples in class C.
MLE variance: σ_C^2 = (1 / n_C) · Σ_{j in class C} (x_j - μ_C)^2 — units: (units of x)^2; this is the MLE variance (biased). Use (1/(n_C-1)) for unbiased sample variance if asked.
Laplace smoothing (add-α): P(x = v | C) = (count_{C,v} + α) / (n_C + α·k) — counts are unitless; α typically 1; k is number of possible values for x (e.g., k=2 for binary).
Worked examples
Example 1 — binary-feature text-like toy (Laplace smoothing): Training data (6 examples):
- Spam (S): 3 examples. Features (offer, money) counts in Spam: offer=2, money=3.
- Ham (H): 3 examples. Features counts in Ham: offer=0, money=1. Task: classify test instance x = (offer=1, money=1). Steps:
- Priors: P(S) = 3/6 = 0.5, P(H) = 3/6 = 0.5.
- Using Laplace α=1, k=2 (binary): P(offer=1 | S) = (2 + 1) / (3 + 2) = 3/5 = 0.6. P(money=1 | S) = (3 + 1) / (3 + 2) = 4/5 = 0.8. P(offer=1 | H) = (0 + 1) / (3 + 2) = 1/5 = 0.2. P(money=1 | H) = (1 + 1) / (3 + 2) = 2/5 = 0.4.
- Posteriors (proportional): Score(S) = P(S)·P(offer=1 | S)·P(money=1 | S) = 0.5·0.6·0.8 = 0.24. Score(H) = 0.5·0.2·0.4 = 0.04.
- Normalise if needed: total = 0.28. P(S | x) = 0.24/0.28 = 0.8571. Answer: choose Spam. Final: P(Spam | x) ≈ 0.857, classify as Spam.
Example 2 — Gaussian naive Bayes (continuous feature): Training: one feature x (seconds). Class A: x = [1.0, 1.2, 0.8]. Class B: x = [2.0, 2.1, 1.9]. Test x0 = 1.1.
- Priors: equal classes → P(A)=0.5, P(B)=0.5.
- MLE means: μ_A = (1.0 + 1.2 + 0.8)/3 = 1.0 (seconds). μ_B = (2.0 + 2.1 + 1.9)/3 = 2.0 (seconds).
- MLE variances: σ_A^2 = [(0)^2 + (0.2)^2 + (-0.2)^2] / 3 = (0 + 0.04 + 0.04)/3 = 0.08/3 = 0.0266667 (s^2). σ_B^2 = [(0)^2 + (0.1)^2 + (-0.1)^2] / 3 = (0 + 0.01 + 0.01)/3 = 0.02/3 = 0.0066667 (s^2).
- Likelihoods using Gaussian formula: P(x0 | A) ≈ (1 / sqrt(2·π·0.0266667)) · exp(- (0.1)^2 / (2·0.0266667)) ≈ 2.443 · exp(-0.1875) ≈ 2.024 (1/s). P(x0 | B) ≈ (1 / sqrt(2·π·0.0066667)) · exp(- (0.9)^2 / (2·0.0066667)) ≈ 4.886 · exp(-60.75) ≈ ~0 (1/s).
- Posterior scores: Score(A)=0.5·2.024≈1.012; Score(B)≈0. Answer: choose class A. Final: class A. (Note densities can exceed 1; units for density are 1/seconds.)
(If implementing, compute log-scores: log 0.5 + log likelihoods, and compare.)
Common mistakes
- Mis-applying Laplace smoothing denominator: remember to add α·k (k = number of possible feature values) to class counts.
- Confusing MLE variance (divide by n) with the unbiased sample variance (divide by n − 1); state which you use.
- Treating the conditional independence assumption as true; in answers, explicitly state it and its consequences.
- Using Gaussian model for categorical data or using discrete counts for continuous features without justification.
For your exam
Typical question types:
- Definitions: state Bayes' theorem, posterior, prior, likelihood, conditional independence, MAP.
- Short notes: explain naive Bayes classifier, its assumptions, advantages and disadvantages. Typical advantages: simple, fast to train and predict, works well with high-dimensional sparse data, robust to many irrelevant features. Typical disadvantages: conditional independence rarely holds, probabilities can be poorly calibrated, zero-frequency and zero-variance issues require smoothing/regularisation.
- Derivations: derive Bayes' theorem from definition of conditional probability; derive naive Bayes posterior under independence assumption.
- Numericals: compute priors, likelihoods, apply Laplace smoothing, classify a test tuple; compute Gaussian parameters and posterior for continuous feature(s).
- Programs / pseudo-code: show training (estimate priors and likelihood parameters) and prediction (compute log-scores and argmax).
- Diagrams: draw probability tree or sample-space rectangle to visualise Bayes' theorem.
What to include in a full-marks answer:
- Clear definitions of terms used.
- Correct statement and derivation of Bayes' theorem.
- Explicit statement of the conditional independence assumption for naive Bayes.
- Correct formulas for likelihoods and parameter estimation, with units where relevant.
- Careful numeric steps, stating whether smoothing is used and which α, and whether MLE or unbiased estimates are used for variance.
Quick check
- State Bayes' theorem in one equation.
- What does the naive Bayes conditional independence assumption say in words?
- Why use log probabilities in implementation? Answers: P(A | B) = P(B | A)·P(A)/P(B).; Features are independent of each other given the class (i.e. P(xi | xj, C) = P(xi | C)).; To turn products into sums and avoid numerical underflow and loss of precision.
Finished this topic? Mark it so your progress, study plan and readiness keep up.
Stuck on something here?