Machine Learning Natural Language Processing Syllabus

What is Part-of-Speech (POS) tagging?

POS tagging assigns each token a grammatical category (such as noun, verb, adjective, adverb, pronoun, preposition) based on its definition and context within the sentence.

Name two classic models used for POS tagging.

Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs); also rule-based taggers and modern neural (BiLSTM) taggers.

In POS tagging, why is context necessary? Give an example.

The same word can have different tags depending on use. E.g. 'book' is a noun in 'read a book' but a verb in 'book a flight'; the surrounding words disambiguate the correct tag.

What is Named Entity Recognition (NER)?

NER is the task of locating and classifying named entities in text into predefined categories such as person, organization, location, date, time, monetary value, and percentage.

What does the BIO (IOB) tagging scheme represent in NER?

It labels tokens as B- (Beginning of an entity), I- (Inside/continuation of an entity), or O (Outside any entity), allowing multi-token entities to be marked, e.g. 'New(B-LOC) York(I-LOC)'.

In the sentence 'Apple was founded in Cupertino in 1976', identify the named entities and their types.

'Apple' = Organization, 'Cupertino' = Location (GPE), '1976' = Date.

What is the Bag of Words (BoW) model?

BoW represents a document as an unordered multiset of its words, recording word counts (or presence) while discarding grammar and word order. Each document becomes a vector over the vocabulary.

State two key limitations of the Bag of Words model.

It ignores word order/context (so 'dog bites man' equals 'man bites dog') and produces high-dimensional sparse vectors that capture no semantic similarity between different words.

What does TF-IDF stand for, and what is its purpose?

Term Frequency–Inverse Document Frequency. It weights terms so that words frequent in a document but rare across the corpus get high scores, highlighting words that are distinctive to a document.

Give the formula for Term Frequency (TF) of term $t$ in document $d$.

$$\text{TF}(t,d) = \frac{f_{t,d}}{\sum_{t' \in d} f_{t',d}}$$ where $f_{t,d}$ is the raw count of term $t$ in document $d$ (sometimes TF is just the raw count $f_{t,d}$).

Give the standard formula for Inverse Document Frequency (IDF) of term $t$.

$$\text{IDF}(t) = \log\frac{N}{n_t}$$ where $N$ is the total number of documents and $n_t$ is the number of documents containing term $t$. A smoothed variant is $\log\frac{N}{1+n_t}$.

Write the full TF-IDF weight of term $t$ in document $d$.

$$\text{TF-IDF}(t,d) = \text{TF}(t,d) \times \text{IDF}(t) = \text{TF}(t,d)\cdot \log\frac{N}{n_t}$$

In TF-IDF, what score does a word that appears in every document receive, and why?

Its IDF is $\log\frac{N}{N} = \log 1 = 0$, so its TF-IDF weight is $0$ — such ubiquitous words are treated as non-discriminative.

What are word embeddings?

Word embeddings are dense, low-dimensional real-valued vector representations of words learned so that semantically/syntactically similar words have nearby vectors, capturing meaning from distributional context.

What is the distributional hypothesis that underlies word embeddings?

'You shall know a word by the company it keeps' — words that occur in similar contexts tend to have similar meanings, so context co-occurrence can encode semantics.

How does the similarity between two word embedding vectors $\vec{a}$ and $\vec{b}$ commonly get measured?

By cosine similarity: $$\cos(\theta) = \frac{\vec{a}\cdot\vec{b}}{\lVert\vec{a}\rVert\,\lVert\vec{b}\rVert}$$ which ranges from $-1$ to $1$, with values near $1$ meaning highly similar.

What famous analogy result demonstrates the linear structure of word embeddings?

$\vec{king} - \vec{man} + \vec{woman} \approx \vec{queen}$ — vector arithmetic on embeddings captures relational/semantic analogies.

What is Word2Vec, and who introduced it?

Word2Vec is a shallow neural network method for learning word embeddings from large corpora, introduced by Tomas Mikolov and colleagues at Google in 2013.

Name and distinguish the two architectures of Word2Vec.

CBOW (Continuous Bag of Words) predicts the target/center word from its surrounding context words; Skip-gram predicts the surrounding context words given the center word. Skip-gram works better for rare words; CBOW is faster.

What training optimizations make Word2Vec efficient?

Negative sampling (only updating a small sample of negative words instead of the full softmax) and hierarchical softmax, which avoid computing the expensive full-vocabulary softmax.

What is GloVe, and how does its approach differ from Word2Vec?

GloVe (Global Vectors), from Stanford (Pennington et al., 2014), learns embeddings by factorizing a global word–word co-occurrence count matrix, using corpus-wide statistics, whereas Word2Vec is a local prediction (window-based) method.

What quantity does GloVe primarily model to learn embeddings?

Ratios of word–word co-occurrence probabilities; its objective relates the dot product of word vectors to the logarithm of their co-occurrence count, $\vec{w_i}\cdot\vec{w_j} \approx \log(X_{ij})$.

What is FastText, and what is its key innovation over Word2Vec?

FastText (Facebook AI, 2016) represents each word as a bag of character n-grams and sums their vectors. This lets it build embeddings for out-of-vocabulary words and capture subword/morphological information.

How does FastText handle out-of-vocabulary (OOV) words, unlike Word2Vec and GloVe?

It composes an OOV word's vector from its character n-gram subword vectors, so even unseen words get meaningful embeddings, whereas Word2Vec/GloVe have no vector for unseen words.

What is an n-gram?

An n-gram is a contiguous sequence of $n$ items (typically words or characters) from a text. For example, in 'natural language processing': unigrams are single words, bigrams are 'natural language' and 'language processing'.

In an n-gram language model, what does the Markov assumption state?

The probability of a word depends only on the previous $n-1$ words: $$P(w_i \mid w_1,\dots,w_{i-1}) \approx P(w_i \mid w_{i-n+1},\dots,w_{i-1})$$

Why is smoothing (e.g. Laplace/add-one) needed in n-gram models?

To assign nonzero probability to n-grams not seen in the training data, avoiding zero probabilities for valid but unobserved word sequences.

What is a Recurrent Neural Network (RNN), and why is it suited to NLP?

An RNN is a neural network that processes sequences one element at a time while maintaining a hidden state that carries information from previous steps, making it suited to variable-length sequential data like text. Its update is $$h_t = \tanh(W_h h_{t-1} + W_x x_t + b)$$

What is the vanishing/exploding gradient problem in RNNs?

During backpropagation through time, gradients can shrink toward $0$ (vanishing) or grow without bound (exploding) over many time steps, making it hard for vanilla RNNs to learn long-range dependencies.

What is an LSTM, and how does it address the RNN limitation?

A Long Short-Term Memory network is a gated RNN variant that uses a cell state plus input, forget, and output gates to regulate information flow, preserving long-range dependencies and mitigating the vanishing gradient problem. The forget gate is $f_t = \sigma(W_f[h_{t-1}, x_t] + b_f)$.