KL Divergence & Mutual Information
AdvancedBy the AiCanCode.org team · How we write and check our content
KL divergence measures how far one probability distribution is from another, and mutual information measures how much knowing one variable tells you about another — tools behind VAEs, RLHF, and feature selection.
Overview
Two more information-theory tools appear across modern AI. KL divergence, D(P‖Q), measures how much information is lost when you approximate a true distribution P with a model distribution Q — it is zero when they match and grows as they diverge. It is not symmetric (P‖Q ≠ Q‖P), which matters in practice. KL shows up as the regularizer in variational autoencoders (keeping the learned latent close to a normal prior) and as the guardrail in RLHF/PPO that stops a fine-tuned LLM from drifting too far from the base model. Mutual information measures how much knowing one variable reduces uncertainty about another — a general (even non-linear) measure of dependence used for feature selection, representation learning, and analysing what a network has captured. Both are close relatives of entropy: KL is the extra cross-entropy over the true entropy, and mutual information is the KL between a joint and the product of its marginals.
KL divergence: distance from truth to approximation
KL(P‖Q) = Σ P·log(P/Q). It is ≥ 0, zero only when P = Q, and asymmetric. Note it equals cross-entropy minus entropy.
import numpy as np
def kl(p, q):
p, q = np.array(p), np.array(q)
return np.sum(p * np.log((p + 1e-12) / (q + 1e-12)))
P = [0.7, 0.2, 0.1] # true
Q1 = [0.6, 0.3, 0.1] # close approximation
Q2 = [0.1, 0.2, 0.7] # poor approximation
print(round(kl(P, Q1), 4)) # small
print(round(kl(P, Q2), 4)) # large
print(round(kl(P, P), 4)) # 0.0 -> identical
print(round(kl(P, Q1), 4) == round(kl(Q1, P), 4)) # False -> asymmetricMutual information: how much X tells you about Y
Mutual information is the KL between the joint distribution and the product of marginals — zero for independent variables, positive when they share information. Great for non-linear feature relevance.
import numpy as np
# Joint P(X,Y); if independent, MI = 0.
joint = np.array([[0.4, 0.1],
[0.1, 0.4]]) # X and Y are correlated here
px = joint.sum(axis=1, keepdims=True)
py = joint.sum(axis=0, keepdims=True)
mi = np.sum(joint * np.log((joint + 1e-12) / (px * py + 1e-12)))
print(round(mi, 4)) # > 0 -> X and Y share informationKey Points to Remember
- 1KL divergence measures how far an approximation Q is from the truth P (≥0, asymmetric)
- 2KL = cross-entropy − entropy; it regularizes VAEs and constrains RLHF fine-tuning
- 3Mutual information measures shared information / dependence, including non-linear
- 4MI = 0 iff variables are independent; used for feature selection & representation analysis
Interview Questions
Sign in to ask AriaWhat does KL divergence measure, and why is it not a true distance?
Where does KL divergence appear in VAEs or RLHF?
How does mutual information differ from correlation?
Ask Aria about KL Divergence & Mutual Information
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to ask a question — Aria answers, and so do other students.