Deep Learning Sequence Models and Recurrent Networks Syllabus

Besides gating, name two architectural strategies to help RNNs capture long-range dependencies.

Skip/residual connections across time steps (shorter gradient paths) and attention mechanisms that create direct connections between distant positions, bypassing the sequential bottleneck entirely.

Describe the encoder-decoder (seq2seq) architecture.

An encoder RNN compresses the input sequence into a context representation (e.g. its final hidden state $c$); a decoder RNN generates the output sequence autoregressively, conditioned on $c$ and its own previously generated tokens.

What is the 'information bottleneck' problem in a basic encoder-decoder?

Compressing the entire source sequence into a single fixed-length context vector forces long inputs through a narrow bottleneck, degrading performance as sequence length grows. Attention was introduced to solve this.

What quantities does an attention mechanism compute (query, key, value framing)?

For query $q$ and keys $k_i$/values $v_i$: scores $e_i = \text{score}(q,k_i)$, weights $\alpha_i=\text{softmax}(e_i)=\frac{\exp(e_i)}{\sum_j \exp(e_j)}$, and context $c=\sum_i \alpha_i v_i$.

Give the formula for scaled dot-product attention.

$\text{Attention}(Q,K,V)=\text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V$, where $d_k$ is the key dimension; dividing by $\sqrt{d_k}$ keeps dot products from growing large and saturating the softmax.

Why is the $\frac{1}{\sqrt{d_k}}$ scaling factor used in dot-product attention?

For large $d_k$, dot products $q\cdot k$ have variance proportional to $d_k$, pushing softmax into regions with tiny gradients. Scaling by $\frac{1}{\sqrt{d_k}}$ normalizes the variance to $\approx 1$, keeping gradients healthy.

Contrast additive (Bahdanau) and multiplicative (Luong) attention scoring.

Additive: $e_i = v^{\top}\tanh(W_1 q + W_2 k_i)$, a small feedforward net. Multiplicative: $e_i = q^{\top} k_i$ (or $q^{\top} W k_i$), a dot product—faster and more memory-efficient, especially with optimized matrix multiplication.

What is self-attention, and how does it differ from encoder-decoder attention?

In self-attention the queries, keys, and values all come from the same sequence, letting each position attend to every other position in that sequence. Encoder-decoder (cross) attention uses decoder queries against encoder keys/values.

What problem does beam search decoding solve, and how?

It approximately finds the highest-probability output sequence (intractable to search exhaustively) by keeping the top-$k$ partial hypotheses (the beam) at each decoding step and expanding only those, balancing search quality and cost.

Contrast greedy decoding with beam search.

Greedy decoding picks the single most probable token at each step ($k=1$), which is locally optimal but can miss globally better sequences. Beam search retains $k>1$ candidates, exploring more of the search space at higher computational cost.

Why does beam search use length normalization, and give a common form.

Raw sequence log-probabilities $\sum_t \log P(y_t\mid\cdot)$ decrease with length, biasing toward short outputs. Length normalization divides by a length penalty, e.g. score $=\frac{1}{L^{\alpha}}\sum_{t=1}^{L}\log P(y_t\mid y_{<t})$ with $\alpha\in(0,1]$.

What happens to beam search behavior as the beam width $k$ increases very large versus $k=1$?

$k=1$ is greedy search. Increasing $k$ explores more hypotheses and can raise likelihood, but beyond a point returns diminish and, in NMT, very large beams often produce shorter, lower-quality translations (the 'beam search curse').

What is teacher forcing in sequence model training?

During training the decoder is fed the ground-truth previous token (rather than its own prediction) as input at each step, so $P(y_t\mid y^{*}_{<t})$ is modeled. This stabilizes and speeds convergence.

What is exposure bias, and how does it relate to teacher forcing?

Exposure bias is the train/test mismatch caused by teacher forcing: at training the model always sees correct history, but at inference it sees its own (possibly wrong) outputs, so errors compound with no learned recovery.

What is scheduled sampling and what problem does it mitigate?

Scheduled sampling gradually replaces ground-truth previous tokens with the model's own predictions during training, according to a decaying probability, to reduce exposure bias from pure teacher forcing and bridge the train/test gap.

What is the core objective of Word2Vec, and name its two model variants.

Word2Vec learns dense word embeddings by predicting words from context. Variants: CBOW (predict the center word from surrounding context) and Skip-gram (predict surrounding context words from the center word).

State the Skip-gram softmax objective and why it is expensive.

$P(w_O\mid w_I)=\frac{\exp(v_{w_O}^{\prime\top} v_{w_I})}{\sum_{w=1}^{V}\exp(v_w^{\prime\top} v_{w_I})}$. The denominator sums over the whole vocabulary $V$, making each update $O(V)$—hence approximations like negative sampling or hierarchical softmax are used.

What is negative sampling in Word2Vec?

Instead of the full softmax, it trains a binary classifier to distinguish the true context word from $k$ randomly sampled 'negative' words, optimizing $\log\sigma(v_{w_O}^{\prime\top} v_{w_I}) + \sum_{i=1}^{k}\mathbb{E}_{w_i\sim P_n}[\log\sigma(-v_{w_i}^{\prime\top} v_{w_I})]$.

How does GloVe differ fundamentally from Word2Vec?

GloVe is count-based: it factorizes the global word-word co-occurrence matrix, fitting embeddings so $w_i^{\top}\tilde{w}_j + b_i + \tilde{b}_j \approx \log X_{ij}$. Word2Vec is prediction-based over local context windows; GloVe uses global corpus statistics directly.

Write GloVe's weighted least-squares objective.

$J=\sum_{i,j=1}^{V} f(X_{ij})\left(w_i^{\top}\tilde{w}_j + b_i + \tilde{b}_j - \log X_{ij}\right)^2$, where $f$ is a weighting function that downweights rare and caps frequent co-occurrences.

What famous property do Word2Vec/GloVe embeddings exhibit via vector arithmetic?

Linear analogies: $\vec{king}-\vec{man}+\vec{woman}\approx\vec{queen}$. Semantic and syntactic relations appear as roughly consistent offset vectors in embedding space.

Why are static embeddings like Word2Vec and GloVe limited for polysemy?

They assign a single fixed vector per word type regardless of context, so 'bank' (river vs. finance) collapses to one representation. Contextual embeddings solve this by producing token-specific vectors.

What is tokenization in NLP, and why is word-level tokenization problematic?

Tokenization splits text into units for the model. Word-level tokenization yields huge vocabularies, cannot handle out-of-vocabulary words, and wastes capacity on rare forms—motivating subword approaches.

Explain Byte-Pair Encoding (BPE) as a subword tokenization method.

BPE starts from characters and iteratively merges the most frequent adjacent symbol pair into a new symbol, repeating for a fixed number of merges. Frequent words become single tokens; rare words decompose into subword pieces, eliminating OOV.

How does WordPiece differ from BPE in choosing merges?

BPE merges the most frequent pair; WordPiece merges the pair that most increases the training-corpus likelihood, i.e. it picks the merge maximizing $\frac{P(ab)}{P(a)P(b)}$ under a language model, rather than raw frequency.

What advantage do subword tokenizers provide over word and character tokenization?

They give a fixed, moderate vocabulary with no true OOV tokens (rare words split into known subwords), while keeping sequences much shorter than character-level tokenization—balancing vocabulary size against sequence length.

What are contextual embeddings, and how do they differ from static embeddings?

Contextual embeddings assign each token a vector that depends on its surrounding sentence, produced by a deep model (e.g. ELMo, BERT). The same word gets different vectors in different contexts, capturing polysemy—unlike one-vector-per-type static embeddings.

How does ELMo produce contextual embeddings?

ELMo runs a deep bidirectional LSTM language model and forms each token's embedding as a learned, task-weighted linear combination of the hidden states across all biLSTM layers, capturing both syntax (lower layers) and semantics (higher layers).

What pretraining objective gives BERT its contextual, bidirectional representations?

Masked Language Modeling (MLM): randomly mask ~15% of input tokens and train the model to predict them from both left and right context, plus (originally) Next Sentence Prediction. This yields deeply bidirectional token embeddings.

Why are contextual embeddings generally preferred over Word2Vec/GloVe for downstream tasks?

They encode word sense in context, capture syntax and long-range dependencies through deep bidirectional processing, and provide transferable pretrained representations that can be fine-tuned—typically outperforming static embeddings on most NLP benchmarks.