KL Divergence & Mutual Information — Concept Check
Questions on KL Divergence & Mutual Information. KL divergence measures how far one probability distribution is from another, and mutual information measures how much knowing one variable tells you about another — tools behind VAEs, RLHF, and feature selection.
Questions in this quiz
8 questions on KL Divergence & Mutual Information — Concept Check. Work through them here, then start the quiz to answer them and see your score.
1.A machine learning model uses KL divergence as a regularizer. What is the purpose of this?
- ATo ensure the model's outputs are symmetric
- BTo minimize the cross-entropy of the model
- CTo maximize the entropy of the learned distribution
- DTo keep the learned distribution close to a prior distribution
2.In a feature selection process, mutual information is used. What does a high mutual information score indicate?
- AThe features have low variance
- BThe features are independent of the target variable
- CThe features provide significant information about the target variable
- DThe features have high entropy
3.Two probability distributions P and Q are compared using KL divergence. If KL(P‖Q) is zero, what does this imply?
- AP and Q are independent
- BP and Q are identical
- CP has more entropy than Q
- DQ is a poor approximation of P
4.A developer claims that KL(P‖Q) is equal to KL(Q‖P). What is the flaw in this statement?
- AKL divergence is asymmetric
- BKL divergence is always positive
- CKL divergence is symmetric
- DKL divergence is always zero
5.In a reinforcement learning model using RLHF, KL divergence is used. What role does it play?
- AIt increases the model's entropy
- BIt prevents the model from drifting too far from the base model
- CIt reduces the model's reward variance
- DIt ensures the model explores new actions
6.A data scientist uses mutual information to analyze feature relevance. What does a mutual information score of zero imply?
- AThe features are highly correlated with the target
- BThe features have high variance
- CThe features are irrelevant to the target variable
- DThe features have high entropy
7.When calculating KL divergence, why is it important to add a small constant (e.g., 1e-12) to probabilities?
- ATo ensure probabilities are symmetric
- BTo minimize the entropy of the distribution
- CTo maximize the KL divergence value
- DTo prevent division by zero errors
8.A joint distribution of two variables X and Y shows a mutual information greater than zero. What does this indicate?
- AX and Y are identically distributed
- BX and Y are independent
- CX and Y have equal variance
- DX and Y share information