The Mathematics Behind Transformers & LLMs — Concept Check
Questions on The Mathematics Behind Transformers & LLMs. A transformer turns tokens into vectors and lets them exchange information through attention — a few matrix multiplies, a softmax, and a dot product — which is the mathematical core of every modern LLM.
Questions in this quiz
8 questions on The Mathematics Behind Transformers & LLMs — Concept Check. Work through them here, then start the quiz to answer them and see your score.
1.A transformer model calculates attention scores using dot products between which components?
- AKeys and values
- BEmbeddings and values
- CQueries and keys
- DQueries and values
2.In a transformer, why is the attention score scaled by 1/√d?
- ATo enhance the similarity between tokens
- BTo reduce computational complexity
- CTo ensure numerical stability
- DTo increase the range of scores
3.A developer notices that the attention weights in a transformer model sum to more than 1. What might be the issue?
- AThe dot product is not computed
- BThe query and key matrices are swapped
- CThe softmax function is not applied correctly
- DThe embeddings are not initialized
4.When computing attention in transformers, what is the role of the softmax function?
- ATo normalize attention scores into a probability distribution
- BTo compute dot products between vectors
- CTo project tokens into embeddings
- DTo scale values for numerical stability
5.A model outputs unexpected results after stacking multiple transformer blocks. What could be a potential cause?
- AIncorrect embedding dimensions
- BImproper normalization between layers
- CUntrained query and key matrices
- DMissing softmax application
6.Which of the following is NOT a component used in the self-attention mechanism of transformers?
- AQuery
- BValue
- CKey
- DBias
7.A software engineer claims that transformers do not use gradient descent for training. What is the flaw in this statement?
- ATransformers use only matrix multiplication
- BTransformers rely solely on dot products
- CTransformers use gradient descent to minimize cross-entropy loss
- DTransformers do not require training
8.What happens if the dot product between queries and keys is not scaled in a transformer?
- AThe model may become numerically unstable
- BThe values will not be weighted
- CThe embeddings will not be vectors
- DThe attention weights will be more accurate