Momentum, RMSProp & Adam — Concept Check
Questions on Momentum, RMSProp & Adam. Modern optimizers speed up gradient descent by remembering past gradients (momentum) and adapting the step size per parameter (RMSProp) — Adam combines both and is the default for training deep nets.
Questions in this quiz
8 questions on Momentum, RMSProp & Adam — Concept Check. Work through them here, then start the quiz to answer them and see your score.
1.A neural network is training slowly and oscillating between loss values. Which optimizer adjustment could help?
- AIncrease the learning rate to speed up convergence
- BDecrease the batch size to reduce oscillations
- CUse plain gradient descent for simplicity
- DSwitch to Adam for adaptive learning rates and momentum
2.During training, you notice that the optimizer is taking large steps in steep directions, causing instability. What should you do?
- ASwitch to plain gradient descent for consistent step sizes
- BUse RMSProp to adaptively scale the learning rate per parameter
- CUse momentum to accumulate past gradients
- DIncrease the learning rate for faster convergence
3.Which of the following best describes the role of momentum in optimization?
- AIt adapts the learning rate per parameter to handle steep and flat regions
- BIt accumulates past gradients to smooth out noise and accelerate descent
- CIt uses a fixed step size for all parameters to ensure consistency
- DIt combines gradient accumulation and adaptive scaling for efficiency
4.A model trained with Adam converges faster than expected. What mechanism in Adam contributes most to this?
- AAdaptive scaling of learning rates per parameter
- BFixed learning rate across parameters
- CBias correction for early steps
- DUse of only the first moment (momentum)
5.Why might someone incorrectly believe that plain gradient descent is faster than momentum-based methods?
- AThey assume fixed step sizes are more predictable
- BThey believe momentum ignores recent gradients
- CThey assume adaptive learning rates slow down training
- DThey think momentum adds unnecessary complexity
6.What is the primary advantage of using Adam over just momentum or RMSProp alone?
- AReduces the need for bias correction
- BUses a simpler algorithm with fewer hyperparameters
- CCombines momentum and adaptive scaling for efficient training
- DProvides a fixed learning rate for all parameters
7.A practitioner switches from RMSProp to Adam and notices improved performance. What feature of Adam likely caused this?
- AHigher default learning rate
- BSimpler implementation and fewer hyperparameters
- CCombination of momentum and adaptive scaling
- DUse of only the second moment (adaptive scaling)
8.What common mistake might lead someone to believe RMSProp and Adam are identical?
- ABoth accumulate past gradients, but Adam also includes momentum
- BBoth are used for deep learning, so they must be the same
- CBoth use bias correction, which is unique to Adam
- DBoth adapt learning rates per parameter, ignoring momentum