Bayes’ Theorem

Overview

Bayes’ theorem is the rule for updating beliefs: posterior ∝ likelihood × prior. It lets you compute P(cause | evidence) from P(evidence | cause), which is exactly the direction you usually can measure but not the one you want. The classic gotcha — a highly accurate medical test can still yield mostly false positives for a rare disease — is pure Bayes, and it trains the intuition that base rates (priors) matter enormously. In ML, Bayes underpins the Naive Bayes classifier, Bayesian inference, probabilistic reasoning, and the mindset that we hold beliefs as distributions and revise them as data arrives. Even when a model is not explicitly "Bayesian", this update-with-evidence framing clarifies what learning is.

The theorem and why base rates dominate

# Disease prevalence 1%; test 99% sensitive, 99% specific.
p_disease = 0.01
p_pos_given_disease = 0.99
p_pos_given_healthy = 0.01

p_pos = (p_pos_given_disease * p_disease +
         p_pos_given_healthy * (1 - p_disease))
p_disease_given_pos = p_pos_given_disease * p_disease / p_pos
print(round(p_disease_given_pos, 3))   # ~0.5 -> only 50% despite "99% accurate"

Naive Bayes: Bayes + independence assumption

import numpy as np

# P(spam | words) ∝ P(spam) * Π P(word | spam)   (naive independence)
p_spam = 0.4
p_words_given_spam = [0.8, 0.7, 0.6]        # per-word likelihoods
p_words_given_ham  = [0.1, 0.2, 0.3]
p_ham = 0.6

spam_score = p_spam * np.prod(p_words_given_spam)
ham_score  = p_ham  * np.prod(p_words_given_ham)
print("predict:", "SPAM" if spam_score > ham_score else "HAM")

Key Points to Remember

• 1Bayes: posterior ∝ likelihood × prior — update belief with evidence
• 2P(A|B) = P(B|A)·P(A) / P(B) flips the conditional you can measure into the one you want
• 3Base rates (priors) can dominate — rare events make "accurate" tests misleading
• 4Naive Bayes = Bayes + conditional independence; a strong, fast baseline