Home/Learn/Math for AI/KL Divergence & Mutual Information

KL Divergence & Mutual Information

Advanced
Information Theory

By the AiCanCode.org team · How we write and check our content

KL divergence measures how far one probability distribution is from another, and mutual information measures how much knowing one variable tells you about another — tools behind VAEs, RLHF, and feature selection.

Overview

Two more information-theory tools appear across modern AI. KL divergence, D(P‖Q), measures how much information is lost when you approximate a true distribution P with a model distribution Q — it is zero when they match and grows as they diverge. It is not symmetric (P‖Q ≠ Q‖P), which matters in practice. KL shows up as the regularizer in variational autoencoders (keeping the learned latent close to a normal prior) and as the guardrail in RLHF/PPO that stops a fine-tuned LLM from drifting too far from the base model. Mutual information measures how much knowing one variable reduces uncertainty about another — a general (even non-linear) measure of dependence used for feature selection, representation learning, and analysing what a network has captured. Both are close relatives of entropy: KL is the extra cross-entropy over the true entropy, and mutual information is the KL between a joint and the product of its marginals.

KL divergence: distance from truth to approximation

KL(P‖Q) = Σ P·log(P/Q). It is ≥ 0, zero only when P = Q, and asymmetric. Note it equals cross-entropy minus entropy.

KL(P‖Q) ≥ 0, zero iff equal, and asymmetric
import numpy as np

def kl(p, q):
    p, q = np.array(p), np.array(q)
    return np.sum(p * np.log((p + 1e-12) / (q + 1e-12)))

P = [0.7, 0.2, 0.1]                 # true
Q1 = [0.6, 0.3, 0.1]                # close approximation
Q2 = [0.1, 0.2, 0.7]               # poor approximation
print(round(kl(P, Q1), 4))          # small
print(round(kl(P, Q2), 4))          # large
print(round(kl(P, P), 4))           # 0.0 -> identical
print(round(kl(P, Q1), 4) == round(kl(Q1, P), 4))  # False -> asymmetric

Mutual information: how much X tells you about Y

Mutual information is the KL between the joint distribution and the product of marginals — zero for independent variables, positive when they share information. Great for non-linear feature relevance.

MI = KL(joint ‖ product of marginals); 0 iff independent
import numpy as np

# Joint P(X,Y); if independent, MI = 0.
joint = np.array([[0.4, 0.1],
                  [0.1, 0.4]])      # X and Y are correlated here
px = joint.sum(axis=1, keepdims=True)
py = joint.sum(axis=0, keepdims=True)
mi = np.sum(joint * np.log((joint + 1e-12) / (px * py + 1e-12)))
print(round(mi, 4))                 # > 0 -> X and Y share information

Key Points to Remember

  • 1KL divergence measures how far an approximation Q is from the truth P (≥0, asymmetric)
  • 2KL = cross-entropy − entropy; it regularizes VAEs and constrains RLHF fine-tuning
  • 3Mutual information measures shared information / dependence, including non-linear
  • 4MI = 0 iff variables are independent; used for feature selection & representation analysis

Interview Questions

Sign in to ask Aria
1

What does KL divergence measure, and why is it not a true distance?

HardGoogle
2

Where does KL divergence appear in VAEs or RLHF?

HardProduct
3

How does mutual information differ from correlation?

HardAmazon

Ask Aria about KL Divergence & Mutual Information

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Sign in to save personal notes for this topic.

Discussion

Sign in to ask a question — Aria answers, and so do other students.

Loading discussion…