Data Science Natural Language Processing (NLP) Syllabus

What does TF-IDF stand for?

Term Frequency–Inverse Document Frequency.

Write the formula for Term Frequency (TF) of term $t$ in document $d$.

$$\text{TF}(t,d) = \frac{f_{t,d}}{\sum_{t' \in d} f_{t',d}}$$ where $f_{t,d}$ is the raw count of term $t$ in document $d$.

Write a common formula for Inverse Document Frequency (IDF) of term $t$.

$$\text{IDF}(t) = \log\!\left(\frac{N}{1 + n_t}\right)$$ where $N$ is the total number of documents and $n_t$ is the number of documents containing $t$.

How is the TF-IDF weight computed and what does a high value indicate?

$\text{TF-IDF}(t,d) = \text{TF}(t,d) \times \text{IDF}(t)$. A high value means the term is frequent in the document but rare across the corpus, making it highly discriminative.

What are word embeddings?

Word embeddings are dense, low-dimensional real-valued vector representations of words learned so that semantically similar words lie close together in the vector space.

Name two classic word-embedding algorithms.

Word2Vec (Skip-gram and CBOW) and GloVe (Global Vectors); fastText is another that uses subword information.

Contrast the CBOW and Skip-gram architectures of Word2Vec.

CBOW predicts the target word from its surrounding context words; Skip-gram predicts the surrounding context words from a given target word. Skip-gram works better for rare words.

What famous vector arithmetic example illustrates that embeddings capture analogies?

$$\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}}$$

What similarity measure is typically used to compare two word embeddings?

Cosine similarity: $$\cos(\theta) = \frac{\vec{a}\cdot\vec{b}}{\lVert\vec{a}\rVert\,\lVert\vec{b}\rVert}$$

What is text classification?

Text classification is the task of assigning predefined category labels to text documents, e.g. spam vs. not spam, topic labeling, or sentiment categories.

Name a simple probabilistic algorithm commonly used as a text-classification baseline and the assumption it makes.

Naive Bayes; it assumes conditional independence of features (words) given the class label.

State Bayes' theorem as used in a Naive Bayes text classifier.

$$P(c \mid d) = \frac{P(d \mid c)\,P(c)}{P(d)}$$ where $c$ is a class and $d$ is a document; classification picks the class maximizing $P(c)\prod_i P(w_i \mid c)$.

What are sequence models in NLP and why are they used?

Sequence models process ordered sequences of tokens, maintaining information about order and context. They are used for tasks like language modeling, translation, and tagging where word order matters.

What is a Recurrent Neural Network (RNN)?

An RNN is a neural network that processes sequences one element at a time, maintaining a hidden state that is updated at each step to carry information from previous tokens.

What problem do vanilla RNNs suffer from with long sequences?

The vanishing (and exploding) gradient problem, which prevents them from learning long-range dependencies.

How do LSTM networks address the limitations of vanilla RNNs?

LSTMs use a cell state and gating mechanisms (input, forget, and output gates) to control information flow, allowing them to retain long-range dependencies and mitigate vanishing gradients.

What are the three gates in an LSTM cell?

The forget gate, the input gate, and the output gate.

How does a GRU differ from an LSTM?

A Gated Recurrent Unit (GRU) is a simplified gated RNN with only two gates (reset and update) and no separate cell state, making it computationally lighter while often performing comparably.

Why are neural networks well suited to NLP compared to classical bag-of-words methods?

They learn dense distributed representations, capture nonlinear patterns and context, share parameters across positions, and reduce manual feature engineering.

What is a Convolutional Neural Network (CNN) used for in NLP?

CNNs apply convolutional filters over sequences of word embeddings to detect local n-gram patterns, useful for tasks like sentence/text classification.

What is the Transformer architecture?

The Transformer is a neural architecture introduced in 'Attention Is All You Need' (2017) that relies entirely on self-attention mechanisms (no recurrence or convolution) to model relationships between all tokens in parallel.

What is the core mechanism that powers Transformers?

Self-attention (scaled dot-product attention), which lets each token attend to and weight every other token in the sequence.

Write the scaled dot-product attention formula.

$$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V$$ where $Q$, $K$, $V$ are query, key, and value matrices and $d_k$ is the key dimension.

Why is the dot product scaled by $\frac{1}{\sqrt{d_k}}$ in attention?

To counteract large dot-product magnitudes for high $d_k$ that would push the softmax into regions with extremely small gradients, stabilizing training.

Why do Transformers require positional encoding?

Because self-attention is order-invariant, positional encodings (e.g. sinusoidal functions) are added to embeddings to inject information about token position in the sequence.

What is transfer learning in NLP?

Transfer learning pretrains a model on a large general corpus to learn language representations, then fine-tunes it on a smaller task-specific dataset, transferring knowledge to improve performance with less data.

Contrast BERT and GPT in terms of their pretraining direction.

BERT is a bidirectional encoder pretrained with masked-language modeling (sees both left and right context). GPT is a unidirectional (autoregressive) decoder pretrained to predict the next token left-to-right.

For a classification model, define precision and recall.

Precision $= \frac{TP}{TP+FP}$ (fraction of predicted positives that are correct); Recall $= \frac{TP}{TP+FN}$ (fraction of actual positives correctly identified).

Write the formula for the F1 score and explain what it balances.

$$F_1 = 2\cdot\frac{\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}}$$ It is the harmonic mean of precision and recall, balancing the two.

What evaluation metric is commonly used for machine translation, and what does a higher value mean?

BLEU (Bilingual Evaluation Understudy) measures n-gram overlap between machine output and reference translations; a higher BLEU score (closer to 1 or 100) indicates closer match to human references.

What is perplexity and how is it used to evaluate language models?

Perplexity measures how well a probability model predicts a sample; for a language model it is $$\text{PP}(W) = P(w_1,\dots,w_N)^{-\frac{1}{N}} = \exp\!\left(-\frac{1}{N}\sum_{i=1}^{N}\ln P(w_i \mid w_{<i})\right).$$ Lower perplexity indicates a better model.