QuizzesKL Divergence & Mutual Information — Concept Check
intermediate
math-for-ai

KL Divergence & Mutual Information — Concept Check

Questions on KL Divergence & Mutual Information. KL divergence measures how far one probability distribution is from another, and mutual information measures how much knowing one variable tells you about another — tools behind VAEs, RLHF, and feature selection.

8 minutes 8 questions Pass at 70%
Time Limit
8 minutes - timer starts when you begin
Change Answers Freely
Revisit and change any answer before submitting
Full Review After Submit
Correct answers and explanations shown for every question
Auto-submit on timeout
Quiz submits automatically when time runs out

Questions in this quiz

8 questions on KL Divergence & Mutual Information — Concept Check. Work through them here, then start the quiz to answer them and see your score.

  1. 1.A machine learning model uses KL divergence as a regularizer. What is the purpose of this?

    • ATo ensure the model's outputs are symmetric
    • BTo minimize the cross-entropy of the model
    • CTo maximize the entropy of the learned distribution
    • DTo keep the learned distribution close to a prior distribution
  2. 2.In a feature selection process, mutual information is used. What does a high mutual information score indicate?

    • AThe features have low variance
    • BThe features are independent of the target variable
    • CThe features provide significant information about the target variable
    • DThe features have high entropy
  3. 3.Two probability distributions P and Q are compared using KL divergence. If KL(P‖Q) is zero, what does this imply?

    • AP and Q are independent
    • BP and Q are identical
    • CP has more entropy than Q
    • DQ is a poor approximation of P
  4. 4.A developer claims that KL(P‖Q) is equal to KL(Q‖P). What is the flaw in this statement?

    • AKL divergence is asymmetric
    • BKL divergence is always positive
    • CKL divergence is symmetric
    • DKL divergence is always zero
  5. 5.In a reinforcement learning model using RLHF, KL divergence is used. What role does it play?

    • AIt increases the model's entropy
    • BIt prevents the model from drifting too far from the base model
    • CIt reduces the model's reward variance
    • DIt ensures the model explores new actions
  6. 6.A data scientist uses mutual information to analyze feature relevance. What does a mutual information score of zero imply?

    • AThe features are highly correlated with the target
    • BThe features have high variance
    • CThe features are irrelevant to the target variable
    • DThe features have high entropy
  7. 7.When calculating KL divergence, why is it important to add a small constant (e.g., 1e-12) to probabilities?

    • ATo ensure probabilities are symmetric
    • BTo minimize the entropy of the distribution
    • CTo maximize the KL divergence value
    • DTo prevent division by zero errors
  8. 8.A joint distribution of two variables X and Y shows a mutual information greater than zero. What does this indicate?

    • AX and Y are identically distributed
    • BX and Y are independent
    • CX and Y have equal variance
    • DX and Y share information