QuizzesEntropy & Cross-Entropy — Concept Check
intermediate
math-for-ai

Entropy & Cross-Entropy — Concept Check

Questions on Entropy & Cross-Entropy. Entropy measures the uncertainty (average surprise) in a distribution, and cross-entropy measures how well a predicted distribution matches the truth — which is exactly the loss used to train classifiers and LLMs.

8 minutes 8 questions Pass at 70%
Time Limit
8 minutes - timer starts when you begin
Change Answers Freely
Revisit and change any answer before submitting
Full Review After Submit
Correct answers and explanations shown for every question
Auto-submit on timeout
Quiz submits automatically when time runs out

Questions in this quiz

8 questions on Entropy & Cross-Entropy — Concept Check. Work through them here, then start the quiz to answer them and see your score.

  1. 1.A model predicts probabilities [0.1, 0.8, 0.1] for classes [A, B, C] and the true class is B. What is the cross-entropy loss?

    • A1.204
    • B0.223
    • C0.693
    • D2.303
  2. 2.A fair coin is tossed. What is the entropy of this distribution?

    • A0.0
    • B0.5
    • C2.0
    • D1.0
  3. 3.A model outputs [0.4, 0.3, 0.3] for a true class at index 1. What does this imply about the model's certainty?

    • AThe model's output indicates zero cross-entropy.
    • BThe model is confident and correct, resulting in low cross-entropy.
    • CThe model is confident but wrong, resulting in high cross-entropy.
    • DThe model is unsure, resulting in higher cross-entropy.
  4. 4.A biased coin shows probabilities [0.99, 0.01]. What is the entropy of this distribution?

    • A0.0
    • B1.0
    • C0.5
    • D0.08
  5. 5.A model predicts [0.8, 0.1, 0.1] for a true class at index 1. What is the cross-entropy loss?

    • A1.204
    • B0.223
    • C0.693
    • D2.303
  6. 6.What happens to entropy as the probability distribution becomes more uniform?

    • AEntropy becomes negative.
    • BEntropy increases to its maximum.
    • CEntropy remains unchanged.
    • DEntropy decreases to zero.
  7. 7.A language model is being fine-tuned. What objective is it minimizing?

    • ACross-entropy
    • BEntropy
    • CMean squared error
    • DHinge loss
  8. 8.Which of the following best describes cross-entropy in classification tasks?

    • AIt measures how well predicted probabilities match true labels.
    • BIt calculates the likelihood of the most probable class.
    • CIt assesses the variance in predicted probabilities.
    • DIt measures the average uncertainty in a single distribution.