Deep Learning Sequence Models and Recurrent Networks Syllabus

Sequence Models and Recurrent Networks syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Sequence Models and Recurrent Networks in Deep Learning, not a summary of it.

1. 1

   Recurrent Neural Networks

   3 topics

   • RNN Architecture
   • Sequence Modeling Tasks
   • Bidirectional RNNs
2. 2

   Gated Recurrent Architectures

   3 topics

   • Long Short-Term Memory (LSTM)
   • Gated Recurrent Units (GRU)
   • Long-Range Dependencies
3. 3

   Sequence-to-Sequence Models

   4 topics

   • Encoder-Decoder Architecture
   • Attention Mechanism
   • Beam Search Decoding
   • Teacher Forcing
4. 4

   Word Embeddings and Text Representation

   3 topics

   • Word2Vec and GloVe
   • Tokenization and Subword Units
   • Contextual Embeddings

Sequence Models and Recurrent Networks flashcards for Deep Learning

24 of 50 cards from the Sequence Models and Recurrent Networks deck — real questions with worked answers.

1. What is the defining recurrence relation of a vanilla (Elman) RNN's hidden state?

   $h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$, with output $y_t = W_{hy} h_t + b_y$. The same weight matrices are shared across all time steps.

2. Why do RNNs share weights across time steps?

   Weight sharing lets the network process sequences of arbitrary length with a fixed number of parameters and enforces the assumption that the same transition dynamics apply at every position (a form of temporal translation invariance).

3. In RNN backpropagation, what is Backpropagation Through Time (BPTT)?

   BPTT unrolls the RNN across time steps into a deep feedforward graph and applies standard backpropagation, summing gradient contributions of the shared weights over all time steps: $\frac{\partial L}{\partial W} = \sum_{t} \frac{\partial L_t}{\partial W}$.

4. What causes the vanishing and exploding gradient problems in RNNs?

   Gradients propagate through repeated multiplication by the recurrent Jacobian. Over $k$ steps the factor behaves like $\prod \|W_{hh}\|$; if the relevant eigenvalues are $<1$ gradients vanish exponentially, if $>1$ they explode, making long-range learning unstable.

5. What is gradient clipping and which RNN problem does it address?

   Gradient clipping rescales the gradient when its norm exceeds a threshold: if $\|g\| > \theta$, set $g \leftarrow \frac{\theta}{\|g\|} g$. It mitigates the exploding gradient problem, not vanishing gradients.

6. Classify the four canonical RNN input/output configurations by shape.

   One-to-one (fixed vector, e.g. plain classification), one-to-many (image captioning), many-to-one (sentiment classification), and many-to-many (sequence labeling if aligned, or encoder-decoder translation if unaligned).

7. What distinguishes a sequence labeling task from a sequence classification task?

   Sequence labeling produces one output per input token (e.g. POS tagging, NER), so outputs are aligned and equal in length to inputs. Sequence classification produces a single label for the whole sequence (e.g. sentiment).

8. What is a language modeling task, and what quantity does it estimate?

   Language modeling estimates the probability of a sequence via the chain rule $P(w_1,\dots,w_T) = \prod_{t=1}^{T} P(w_t \mid w_{1:t-1})$. Each step predicts the next token given the preceding context.

9. How is perplexity defined and what does it measure for a language model?

   $\text{PPL} = \exp\!\left(-\frac{1}{T}\sum_{t=1}^{T} \log P(w_t \mid w_{1:t-1})\right)$. It is the exponentiated average negative log-likelihood; lower perplexity means the model assigns higher probability to the held-out text.

10. How does a bidirectional RNN compute its representation at each position?

    It runs a forward RNN and an independent backward RNN, then concatenates their hidden states: $h_t = [\overrightarrow{h}_t ; \overleftarrow{h}_t]$, so each position sees both past and future context.

11. What is the main limitation of bidirectional RNNs regarding usage?

    They require the entire input sequence to be available before producing outputs, so they cannot be used for real-time/online generation or autoregressive next-token prediction—only for tasks where the full sequence is known (e.g. tagging, encoding).

12. List the three gates of an LSTM cell and what each controls.

    Forget gate $f_t$ (how much of the previous cell state to keep), input gate $i_t$ (how much new candidate information to write), and output gate $o_t$ (how much of the cell state to expose as the hidden state).

13. Write the full set of LSTM equations.

    $f_t=\sigma(W_f[h_{t-1},x_t]+b_f)$, $i_t=\sigma(W_i[h_{t-1},x_t]+b_i)$, $o_t=\sigma(W_o[h_{t-1},x_t]+b_o)$, $\tilde{c}_t=\tanh(W_c[h_{t-1},x_t]+b_c)$, $c_t=f_t\odot c_{t-1}+i_t\odot\tilde{c}_t$, $h_t=o_t\odot\tanh(c_t)$.

14. Why does the LSTM cell state help preserve long-range gradients?

    The cell-state update $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$ is additive (a near-linear 'constant error carousel'). When $f_t \approx 1$, gradients flow through $c_t$ with minimal decay, avoiding the multiplicative vanishing seen in vanilla RNNs.

15. What are the two roles of the LSTM's hidden state $h_t$ versus cell state $c_t$?

    $c_t$ is the internal long-term memory carried mostly linearly through time; $h_t = o_t \odot \tanh(c_t)$ is the gated, exposed output used for predictions and passed to the next step's gates.

16. List the gates of a GRU and their roles.

    Update gate $z_t$ (interpolates between previous hidden state and new candidate) and reset gate $r_t$ (controls how much past state influences the candidate). GRU has no separate cell state or output gate.

17. Write the GRU equations.

    $z_t=\sigma(W_z[h_{t-1},x_t])$, $r_t=\sigma(W_r[h_{t-1},x_t])$, $\tilde{h}_t=\tanh(W[r_t\odot h_{t-1},x_t])$, $h_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde{h}_t$.

18. Compare GRU and LSTM in structure and parameter count.

    GRU has 2 gates and merges cell and hidden state into one vector; LSTM has 3 gates plus a separate cell state. GRU has roughly 3/4 the parameters of an LSTM per unit, trains faster, and often performs comparably; LSTM can be stronger on very long or complex sequences.

19. In the GRU update $h_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde{h}_t$, what does $z_t \approx 0$ imply?

    When $z_t \approx 0$ the unit copies the previous hidden state almost unchanged ($h_t \approx h_{t-1}$), letting information persist over many steps and preserving gradient flow for long-range dependencies.

20. What is a 'long-range dependency' and why is it hard for vanilla RNNs?

    It is a statistical dependency between elements far apart in a sequence (e.g. subject-verb agreement across a long clause). Vanilla RNNs struggle because gradients over many steps vanish/explode, so the influence of distant inputs is lost.

21. Besides gating, name two architectural strategies to help RNNs capture long-range dependencies.

    Skip/residual connections across time steps (shorter gradient paths) and attention mechanisms that create direct connections between distant positions, bypassing the sequential bottleneck entirely.

22. Describe the encoder-decoder (seq2seq) architecture.

    An encoder RNN compresses the input sequence into a context representation (e.g. its final hidden state $c$); a decoder RNN generates the output sequence autoregressively, conditioned on $c$ and its own previously generated tokens.

23. What is the 'information bottleneck' problem in a basic encoder-decoder?

    Compressing the entire source sequence into a single fixed-length context vector forces long inputs through a narrow bottleneck, degrading performance as sequence length grows. Attention was introduced to solve this.

24. What quantities does an attention mechanism compute (query, key, value framing)?

    For query $q$ and keys $k_i$/values $v_i$: scores $e_i = \text{score}(q,k_i)$, weights $\alpha_i=\text{softmax}(e_i)=\frac{\exp(e_i)}{\sum_j \exp(e_j)}$, and context $c=\sum_i \alpha_i v_i$.

See more Sequence Models and Recurrent Networks flashcards →

Planning Sequence Models and Recurrent Networks for Deep Learning

Sequence Models and Recurrent Networks (Deep Learning) FAQ