Deep Learning Transformers and Attention Syllabus

Transformers and Attention syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Transformers and Attention in Deep Learning, not a summary of it.

1. 1

   The Transformer Architecture

   4 topics

   • Self-Attention
   • Multi-Head Attention
   • Positional Encoding
   • Encoder and Decoder Stacks
2. 2

   Pretrained Language Models

   4 topics

   • BERT and Masked Language Modeling
   • GPT and Autoregressive Models
   • T5 and Encoder-Decoder Models
   • Tokenizers (BPE, WordPiece, SentencePiece)
3. 3

   Large Language Models

   5 topics

   • Scaling Laws and Emergent Abilities
   • In-Context Learning and Prompting
   • Fine-Tuning and Alignment
   • Parameter-Efficient Fine-Tuning
   • Retrieval-Augmented Generation
4. 4

   Vision and Multimodal Transformers

   3 topics

   • Vision Transformers (ViT)
   • CLIP and Contrastive Pretraining
   • Multimodal Architectures
5. 5

   Efficient Attention

   4 topics

   • FlashAttention
   • Sparse and Linear Attention
   • KV Caching
   • Mixture of Experts

Transformers and Attention flashcards for Deep Learning

20 of 58 cards from the Transformers and Attention deck — real questions with worked answers.

1. What is self-attention, and what three vectors are computed for each input token?

   Self-attention lets each token attend to all tokens in the sequence to build context-aware representations. For each token it computes a Query ($Q$), Key ($K$), and Value ($V$) vector by multiplying the input embedding with learned weight matrices $W_Q$, $W_K$, $W_V$.

2. Write the scaled dot-product attention formula.

   $$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^{T}}{\sqrt{d_k}}\right)V$$ where $d_k$ is the dimension of the key vectors.

3. Why is the dot product scaled by $\frac{1}{\sqrt{d_k}}$ in attention?

   For large $d_k$, the dot products grow large in magnitude, pushing the softmax into regions with extremely small gradients. Dividing by $\sqrt{d_k}$ keeps the variance around 1, stabilizing gradients and training.

4. What is the time and memory complexity of standard self-attention with respect to sequence length $n$?

   $O(n^{2}\,d)$ time and $O(n^{2})$ memory, because every token attends to every other token, producing an $n \times n$ attention matrix.

5. What is multi-head attention and why is it used?

   Multi-head attention runs $h$ separate attention operations (heads) in parallel on different learned linear projections of $Q$, $K$, $V$, then concatenates and projects the results. It lets the model jointly attend to information from different representation subspaces and positions.

6. Give the formula for multi-head attention.

   $$\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1,\dots,\text{head}_h)W^{O}$$ where $\text{head}_i = \text{Attention}(QW_i^{Q}, KW_i^{K}, VW_i^{V})$.

7. In the original Transformer, with model dimension $d_{model}=512$ and $h=8$ heads, what is the dimension per head?

   Each head uses $d_k = d_v = \frac{d_{model}}{h} = \frac{512}{8} = 64$. This keeps total computation similar to single-head attention at full dimensionality.

8. Why do Transformers need positional encoding?

   Self-attention is permutation-invariant and has no inherent notion of token order. Positional encodings inject information about the absolute or relative position of tokens so the model can use sequence order.

9. Write the sinusoidal positional encoding formulas from the original Transformer.

   $$PE_{(pos,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d_{model}}}\right), \quad PE_{(pos,2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$ where $pos$ is the position and $i$ is the dimension index.

10. Contrast absolute, learned, and relative (e.g., RoPE) positional encodings.

    Absolute (sinusoidal) uses fixed functions of position; learned encodings train a position embedding table; relative encodings (e.g., RoPE, ALiBi) encode the distance between tokens rather than absolute positions, generalizing better to longer sequences.

11. What are the two main sublayers in each Transformer encoder block, and how are they wrapped?

    Each encoder block has (1) a multi-head self-attention sublayer and (2) a position-wise feed-forward network. Each sublayer is wrapped with a residual (skip) connection followed by layer normalization: $\text{LayerNorm}(x + \text{Sublayer}(x))$.

12. How does a Transformer decoder block differ from an encoder block?

    A decoder block has three sublayers: (1) masked multi-head self-attention (causal masking prevents attending to future tokens), (2) cross-attention over the encoder outputs, and (3) a feed-forward network, each with residual connections and layer norm.

13. What is the position-wise feed-forward network in a Transformer, and its typical hidden size?

    It is applied identically to each position: $$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$$ Typically the inner dimension is $4\times$ the model dimension (e.g., $2048$ for $d_{model}=512$).

14. What is causal (autoregressive) masking in decoder self-attention?

    A mask sets the attention scores for all future positions to $-\infty$ before the softmax, so position $i$ can only attend to positions $\leq i$. This preserves the autoregressive property during training and generation.

15. Compare Pre-LN and Post-LN Transformer architectures.

    Post-LN (original) applies LayerNorm after the residual addition; it can be unstable and needs warmup. Pre-LN applies LayerNorm inside the residual branch before the sublayer, giving more stable gradients and easier training of deep models, at a slight quality cost.

16. What is BERT and what is its architectural type?

    BERT (Bidirectional Encoder Representations from Transformers) is an encoder-only Transformer that produces bidirectional contextual representations. It is pretrained then fine-tuned for downstream tasks like classification and question answering.

17. Describe the Masked Language Modeling (MLM) objective used to pretrain BERT.

    Randomly mask ~15% of input tokens and train the model to predict the original tokens from bidirectional context. Of the chosen tokens, 80% are replaced with [MASK], 10% with a random token, and 10% kept unchanged.

18. What was BERT's second pretraining objective, and what did later work find about it?

    Next Sentence Prediction (NSP): predict whether sentence B follows sentence A. Later work (e.g., RoBERTa) found NSP largely unhelpful and removed it, training only with MLM on longer sequences.

19. What special tokens does BERT use and what are their roles?

    [CLS] is prepended and its final hidden state is used for classification/pooling; [SEP] separates sentence segments and marks the end; [MASK] marks masked tokens; segment embeddings distinguish sentence A from B.

20. What is GPT and what pretraining objective does it use?

    GPT (Generative Pretrained Transformer) is a decoder-only autoregressive model trained with causal language modeling: predict the next token given all previous tokens, maximizing $$\sum_{t} \log P(x_t \mid x_{<t}).$$

See more Transformers and Attention flashcards →

Planning Transformers and Attention for Deep Learning

Transformers and Attention (Deep Learning) FAQ