Information Theory — Cheat Sheet

Entropy measures the uncertainty (average surprise) in a distribution, and cross-entropy measures how well a predicted distribution matches the truth — which is exactly the loss used to train classifiers and LLMs.

• ✓Surprise of an outcome = −log(probability); rare = surprising
• ✓Entropy = average surprise; maximal for uniform, zero for certain outcomes
• ✓Cross-entropy measures the mismatch between true and predicted distributions
• ✓Minimising cross-entropy is the standard loss for classification and next-token LLM training

import numpy as np

def entropy(p):
    p = np.array(p)
    return -np.sum(p * np.log2(p + 1e-12))    # in bits

print(round(entropy([0.5, 0.5]), 3))   # 1.0  -> fair coin, max uncertainty
print(round(entropy([0.99, 0.01]), 3)) # 0.08 -> nearly certain, low entropy
print(round(entropy([1.0, 0.0]), 3))   # 0.0  -> no surprise at all

KL divergence measures how far one probability distribution is from another, and mutual information measures how much knowing one variable tells you about another — tools behind VAEs, RLHF, and feature selection.

• ✓KL divergence measures how far an approximation Q is from the truth P (≥0, asymmetric)
• ✓KL = cross-entropy − entropy; it regularizes VAEs and constrains RLHF fine-tuning
• ✓Mutual information measures shared information / dependence, including non-linear
• ✓MI = 0 iff variables are independent; used for feature selection & representation analysis

import numpy as np

def kl(p, q):
    p, q = np.array(p), np.array(q)
    return np.sum(p * np.log((p + 1e-12) / (q + 1e-12)))

P = [0.7, 0.2, 0.1]                 # true
Q1 = [0.6, 0.3, 0.1]                # close approximation
Q2 = [0.1, 0.2, 0.7]               # poor approximation
print(round(kl(P, Q1), 4))          # small
print(round(kl(P, Q2), 4))          # large
print(round(kl(P, P), 4))           # 0.0 -> identical
print(round(kl(P, Q1), 4) == round(kl(Q1, P), 4))  # False -> asymmetric