Data Science & Machine Learning Deep Learning and Natural Language Processing Syllabus

What is the Adam optimizer?

Adam (Adaptive Moment Estimation) is an optimization algorithm that combines momentum and adaptive per-parameter learning rates using estimates of first and second moments of gradients. It is widely used for fast, robust convergence.

What is TensorFlow?

An open-source deep learning framework developed by Google for building and training machine learning models, using computational graphs and tensors, with support for CPUs, GPUs, and TPUs.

What is Keras and its relationship to TensorFlow?

Keras is a high-level neural network API focused on ease of use. It is the official high-level API integrated into TensorFlow (tf.keras), allowing models to be built with minimal code.

Name the three ways to build a model in Keras.

The Sequential API (linear stack of layers), the Functional API (for complex, multi-input/output or non-linear topologies), and Model subclassing (custom, fully flexible models).

In Keras, what do the compile() and fit() methods do?

compile() configures the model with an optimizer, loss function, and metrics. fit() trains the model on the data for a specified number of epochs and batch size.

What is PyTorch?

An open-source deep learning framework developed by Meta (Facebook) known for its dynamic computation graphs, Pythonic interface, and strong research community adoption.

What is a tensor in PyTorch?

A multi-dimensional array (similar to a NumPy array) that is the core data structure in PyTorch, supporting GPU acceleration and automatic differentiation.

What is autograd in PyTorch?

PyTorch's automatic differentiation engine that records operations on tensors (with requires_grad=True) and automatically computes gradients during the backward pass via .backward().

Compare PyTorch's and TensorFlow's computation graphs.

PyTorch uses dynamic computation graphs (define-by-run, built at runtime), favored for flexibility and debugging. TensorFlow traditionally used static graphs (define-then-run), though TF 2.x adopted eager execution by default.

In PyTorch, what is the typical training loop sequence per batch?

1) optimizer.zero_grad() to clear gradients, 2) forward pass to get predictions, 3) compute loss, 4) loss.backward() to compute gradients, 5) optimizer.step() to update weights.

What is a Convolutional Neural Network (CNN) primarily used for?

CNNs are primarily used for processing grid-like data such as images, performing tasks like image classification, object detection, and segmentation by automatically learning spatial feature hierarchies.

What does a convolutional layer do in a CNN?

It applies learnable filters (kernels) that slide over the input, computing dot products to produce feature maps that detect local patterns like edges, textures, and shapes.

What is a pooling layer in a CNN and name a common type?

A pooling layer downsamples feature maps to reduce spatial dimensions and computation while retaining important features and adding translation invariance. Max pooling (taking the maximum in each region) is common.

What are stride and padding in a convolution?

Stride is the number of pixels the filter moves each step (larger stride = smaller output). Padding adds extra pixels (often zeros) around the input border to control output size and preserve edge information.

Why do CNNs use parameter sharing?

The same filter weights are reused across all spatial positions of the input. This drastically reduces the number of parameters compared to fully connected layers and helps detect features regardless of location.

What is a Recurrent Neural Network (RNN) and what data is it suited for?

An RNN is a network with loops that maintain a hidden state (memory) across time steps, making it suited for sequential data such as text, time series, and speech.

What problem do RNNs commonly suffer from with long sequences?

Vanishing and exploding gradients, which make it difficult to learn long-range dependencies across many time steps.

What is an LSTM and how does it improve on vanilla RNNs?

Long Short-Term Memory is an RNN variant with a cell state and gates (input, forget, output) that regulate information flow, allowing it to capture long-term dependencies and mitigate vanishing gradients.

What are the three gates in an LSTM and their roles?

Forget gate (decides what to discard from cell state), input gate (decides what new information to store), and output gate (decides what to output from the cell state).

What is a GRU and how does it compare to an LSTM?

A Gated Recurrent Unit is a simplified RNN variant with two gates (reset and update) and no separate cell state. It is computationally lighter and faster than an LSTM while often achieving comparable performance.

What is text preprocessing in NLP?

The set of steps that clean and standardize raw text into a usable form for models, including tokenization, lowercasing, removing punctuation/stop words, stemming, and lemmatization.

What is tokenization?

The process of breaking text into smaller units called tokens, such as words, subwords, or characters, which serve as the basic input units for NLP models.

Compare stemming and lemmatization.

Stemming crudely chops word endings to a root (e.g., 'studies' -> 'studi'), fast but may produce non-words. Lemmatization uses vocabulary and morphology to return a valid base form (lemma) (e.g., 'studies' -> 'study'), more accurate but slower.

What are stop words and why are they often removed?

Stop words are common words (e.g., 'the', 'is', 'and') that carry little semantic meaning. They are often removed to reduce noise and dimensionality, though some tasks retain them.

What is the Bag-of-Words (BoW) model?

A text representation that counts word occurrences in a document while ignoring grammar and word order, producing a vector of word frequencies over the vocabulary.

What is TF-IDF and what does it measure?

Term Frequency-Inverse Document Frequency weights a word by how often it appears in a document (TF) times how rare it is across all documents (IDF). It highlights words that are important to a specific document but uncommon overall.

What are word embeddings?

Dense, low-dimensional vector representations of words that capture semantic meaning, such that words with similar meanings have similar vectors. Examples include Word2Vec and GloVe.

Compare one-hot encoding with word embeddings.

One-hot encoding produces sparse, high-dimensional vectors with no semantic relationships (every word equidistant). Word embeddings are dense, low-dimensional, and capture semantic similarity between words.

Name three common NLP applications.

Sentiment analysis, machine translation, named entity recognition (others include text summarization, question answering, chatbots, and spam detection).

What is the Transformer architecture and its key innovation?

A neural network architecture (introduced in 'Attention Is All You Need') that relies entirely on self-attention mechanisms instead of recurrence or convolution, enabling parallel processing of sequences and effective modeling of long-range dependencies. It is the basis for models like BERT and GPT.