Entropy & Cross-Entropy

Overview

Information theory gives AI its favourite loss function, so it is worth the intuition. Entropy is the average "surprise" of outcomes from a distribution: a fair coin (50/50) has high entropy because you genuinely cannot predict it, while a biased coin (99/1) has low entropy because the outcome is nearly certain. Surprise is measured as −log(probability): rare events are surprising, certain events are not. Cross-entropy extends this to two distributions — the true labels and the model's predictions — and measures how many "bits of surprise" you incur by using the model's probabilities to describe reality. Minimising cross-entropy pushes the model to put high probability on the correct class; it is the standard classification loss and the exact objective that trains language models to predict the next token. Every time you fine-tune an LLM, you are minimising cross-entropy.

Entropy: how uncertain is this distribution?

import numpy as np

def entropy(p):
    p = np.array(p)
    return -np.sum(p * np.log2(p + 1e-12))    # in bits

print(round(entropy([0.5, 0.5]), 3))   # 1.0  -> fair coin, max uncertainty
print(round(entropy([0.99, 0.01]), 3)) # 0.08 -> nearly certain, low entropy
print(round(entropy([1.0, 0.0]), 3))   # 0.0  -> no surprise at all

Cross-entropy: the classification & LLM loss

import numpy as np

def cross_entropy(true_onehot, pred):
    return -np.sum(np.array(true_onehot) * np.log(np.array(pred) + 1e-12))

true = [0, 1, 0]                          # correct class = index 1
print(round(cross_entropy(true, [0.1, 0.8, 0.1]), 3))  # 0.223  confident+correct -> low
print(round(cross_entropy(true, [0.4, 0.3, 0.3]), 3))  # 1.204  unsure -> higher
print(round(cross_entropy(true, [0.8, 0.1, 0.1]), 3))  # 2.303  confident+WRONG -> high

Key Points to Remember

• 1Surprise of an outcome = −log(probability); rare = surprising
• 2Entropy = average surprise; maximal for uniform, zero for certain outcomes
• 3Cross-entropy measures the mismatch between true and predicted distributions
• 4Minimising cross-entropy is the standard loss for classification and next-token LLM training