KL Divergence & Mutual Information

Overview

Two more information-theory tools appear across modern AI. KL divergence, D(P‖Q), measures how much information is lost when you approximate a true distribution P with a model distribution Q — it is zero when they match and grows as they diverge. It is not symmetric (P‖Q ≠ Q‖P), which matters in practice. KL shows up as the regularizer in variational autoencoders (keeping the learned latent close to a normal prior) and as the guardrail in RLHF/PPO that stops a fine-tuned LLM from drifting too far from the base model. Mutual information measures how much knowing one variable reduces uncertainty about another — a general (even non-linear) measure of dependence used for feature selection, representation learning, and analysing what a network has captured. Both are close relatives of entropy: KL is the extra cross-entropy over the true entropy, and mutual information is the KL between a joint and the product of its marginals.

KL divergence: distance from truth to approximation

import numpy as np

def kl(p, q):
    p, q = np.array(p), np.array(q)
    return np.sum(p * np.log((p + 1e-12) / (q + 1e-12)))

P = [0.7, 0.2, 0.1]                 # true
Q1 = [0.6, 0.3, 0.1]                # close approximation
Q2 = [0.1, 0.2, 0.7]               # poor approximation
print(round(kl(P, Q1), 4))          # small
print(round(kl(P, Q2), 4))          # large
print(round(kl(P, P), 4))           # 0.0 -> identical
print(round(kl(P, Q1), 4) == round(kl(Q1, P), 4))  # False -> asymmetric

Mutual information: how much X tells you about Y

import numpy as np

# Joint P(X,Y); if independent, MI = 0.
joint = np.array([[0.4, 0.1],
                  [0.1, 0.4]])      # X and Y are correlated here
px = joint.sum(axis=1, keepdims=True)
py = joint.sum(axis=0, keepdims=True)
mi = np.sum(joint * np.log((joint + 1e-12) / (px * py + 1e-12)))
print(round(mi, 4))                 # > 0 -> X and Y share information

Key Points to Remember

• 1KL divergence measures how far an approximation Q is from the truth P (≥0, asymmetric)
• 2KL = cross-entropy − entropy; it regularizes VAEs and constrains RLHF fine-tuning
• 3Mutual information measures shared information / dependence, including non-linear
• 4MI = 0 iff variables are independent; used for feature selection & representation analysis