Entropy & Cross-Entropy — Concept Check
Questions on Entropy & Cross-Entropy. Entropy measures the uncertainty (average surprise) in a distribution, and cross-entropy measures how well a predicted distribution matches the truth — which is exactly the loss used to train classifiers and LLMs.
Questions in this quiz
8 questions on Entropy & Cross-Entropy — Concept Check. Work through them here, then start the quiz to answer them and see your score.
1.A model predicts probabilities [0.1, 0.8, 0.1] for classes [A, B, C] and the true class is B. What is the cross-entropy loss?
- A1.204
- B0.223
- C0.693
- D2.303
2.A fair coin is tossed. What is the entropy of this distribution?
- A0.0
- B0.5
- C2.0
- D1.0
3.A model outputs [0.4, 0.3, 0.3] for a true class at index 1. What does this imply about the model's certainty?
- AThe model's output indicates zero cross-entropy.
- BThe model is confident and correct, resulting in low cross-entropy.
- CThe model is confident but wrong, resulting in high cross-entropy.
- DThe model is unsure, resulting in higher cross-entropy.
4.A biased coin shows probabilities [0.99, 0.01]. What is the entropy of this distribution?
- A0.0
- B1.0
- C0.5
- D0.08
5.A model predicts [0.8, 0.1, 0.1] for a true class at index 1. What is the cross-entropy loss?
- A1.204
- B0.223
- C0.693
- D2.303
6.What happens to entropy as the probability distribution becomes more uniform?
- AEntropy becomes negative.
- BEntropy increases to its maximum.
- CEntropy remains unchanged.
- DEntropy decreases to zero.
7.A language model is being fine-tuned. What objective is it minimizing?
- ACross-entropy
- BEntropy
- CMean squared error
- DHinge loss
8.Which of the following best describes cross-entropy in classification tasks?
- AIt measures how well predicted probabilities match true labels.
- BIt calculates the likelihood of the most probable class.
- CIt assesses the variance in predicted probabilities.
- DIt measures the average uncertainty in a single distribution.