Probability Rules & Conditional Probability

Overview

Machine learning is reasoning under uncertainty, and probability is its language. A probability is a number in [0,1]; the probabilities of all mutually exclusive outcomes sum to 1. Two rules do most of the work: the sum rule (probability of A or B) and the product rule (probability of A and B). The key concept is conditional probability, P(A|B) — the probability of A given that B has happened — because prediction is exactly this: P(label | features). Independence (when knowing B tells you nothing about A, so P(A|B)=P(A)) is the simplifying assumption behind the Naive Bayes classifier and shows up whenever we factorise complex joint distributions. Getting comfortable with "update the probability once you condition on evidence" is the mental move that unlocks Bayesian thinking and generative models.

Conditional probability from counts

import numpy as np

# columns: [is_spam, has_word_free]
data = np.array([[1,1],[1,1],[1,0],[0,0],[0,1],[0,0],[0,0],[1,1]])
spam, free = data[:,0], data[:,1]

p_free = free.mean()
p_spam_and_free = ((spam==1) & (free==1)).mean()
p_spam_given_free = p_spam_and_free / p_free        # P(spam | free)
print(round(p_spam_given_free, 3))   # higher than base spam rate

Independence and the chain rule of probability

import numpy as np

# Independent: P(A and B) = P(A) * P(B)
pA, pB = 0.3, 0.5
print("indep joint:", pA * pB)     # 0.15

# Chain rule (general): P(w1,w2,w3) = P(w1) P(w2|w1) P(w3|w1,w2)
# This is exactly how an LLM assigns probability to a sentence.
p = 0.4 * 0.6 * 0.7
print("sentence prob:", round(p, 3))

Key Points to Remember

• 1A probability lies in [0,1]; exclusive outcomes sum to 1
• 2Conditional P(A|B) = P(A and B) / P(B): belief updated by evidence
• 3Prediction is conditional probability: P(label | features)
• 4Independence lets joints factor into products; the chain rule factors them into conditionals (LLMs)