Data Science & Machine Learning Deep Learning and Natural Language Processing Syllabus

Deep Learning and Natural Language Processing syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Deep Learning and Natural Language Processing in Data Science & Machine Learning, not a summary of it.

1. 1

   Neural Network Fundamentals

   3 topics

   • Perceptron and Activation Functions
     • ReLU, sigmoid and tanh
     • Softmax output
   • Forward and Backpropagation
     • Loss functions
     • Gradient descent variants
   • Training Considerations
     • Learning rate and epochs
     • Dropout and regularization
     • Batch normalization
2. 2

   Deep Learning Frameworks

   4 topics

   • TensorFlow and Keras
     • Sequential and functional API
     • Model compilation and fit
   • PyTorch Basics
     • Tensors and autograd
   • Convolutional Neural Networks
     • Convolution and pooling layers
     • Image classification
   • Recurrent Networks
     • RNN and LSTM
     • Sequence modeling
3. 3

   Natural Language Processing

   4 topics

   • Text Preprocessing
     • Tokenization and stopwords
     • Stemming and lemmatization
   • Text Representation
     • Bag of Words and TF-IDF
     • Word embeddings
   • NLP Applications
     • Sentiment analysis
     • Text classification
   • Transformers Overview
     • Attention mechanism
     • Pretrained language models

Deep Learning and Natural Language Processing flashcards for Data Science & Machine Learning

21 of 50 cards from the Deep Learning and Natural Language Processing deck — real questions with worked answers.

1. What is a perceptron in neural networks?

   A single artificial neuron that computes a weighted sum of inputs plus a bias, then applies an activation function to produce an output. It is the simplest building block of a neural network and is a linear binary classifier.

2. What is the mathematical formula for a perceptron's output before activation?

   z = (w1*x1 + w2*x2 + ... + wn*xn) + b, i.e. z = w·x + b, where w are weights, x are inputs, and b is the bias.

3. Why are non-linear activation functions necessary in neural networks?

   Without non-linearity, stacking layers would still produce only a linear function, so the network could not learn complex, non-linear patterns. Non-linear activations let the network approximate arbitrary functions.

4. Define the sigmoid activation function and its output range.

   Sigmoid: σ(x) = 1 / (1 + e^(-x)). It maps any real input to the range (0, 1), making it useful for representing probabilities.

5. What is the formula and output range of the tanh activation function?

   tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x)). It outputs values in the range (-1, 1) and is zero-centered.

6. Define the ReLU activation function.

   ReLU (Rectified Linear Unit): f(x) = max(0, x). It outputs x if x > 0 and 0 otherwise. It is computationally cheap and helps mitigate the vanishing gradient problem.

7. What is the 'dying ReLU' problem and one solution?

   Neurons can get stuck outputting 0 for all inputs (zero gradient, never updating) when inputs are consistently negative. Leaky ReLU, which allows a small negative slope (f(x) = max(αx, x)), helps prevent this.

8. What does the softmax function do and where is it typically used?

   Softmax converts a vector of raw scores into a probability distribution that sums to 1: softmax(xi) = e^(xi) / Σ e^(xj). It is typically used in the output layer for multi-class classification.

9. What is forward propagation in a neural network?

   The process of passing input data through the network layer by layer, computing weighted sums and activations at each layer, to produce the final output (prediction).

10. What is backpropagation?

    An algorithm that computes gradients of the loss function with respect to each weight by applying the chain rule backward through the network, enabling weight updates via gradient descent.

11. Which mathematical rule is the foundation of backpropagation?

    The chain rule of calculus, which lets the gradient of the loss be decomposed into a product of partial derivatives propagated backward through each layer.

12. What is the gradient descent weight update rule?

    w := w - η * (∂L/∂w), where η is the learning rate, L is the loss, and ∂L/∂w is the gradient of the loss with respect to the weight.

13. What role does the learning rate play in training?

    It controls the step size for weight updates. Too high can cause divergence or overshooting; too low causes slow convergence or getting stuck in local minima.

14. Compare batch, mini-batch, and stochastic gradient descent.

    Batch GD uses the entire dataset per update (stable but slow). Stochastic GD (SGD) uses one sample per update (fast but noisy). Mini-batch GD uses a small subset per update, balancing speed and stability; it is the most common.

15. What is the vanishing gradient problem?

    In deep networks, gradients can become extremely small as they propagate backward through many layers, causing early layers to learn very slowly or stop learning. It is common with sigmoid/tanh activations.

16. What is overfitting and what are two common ways to reduce it?

    Overfitting is when a model learns training data too well (including noise) and generalizes poorly to new data. It can be reduced via regularization (L1/L2, dropout) and gathering more/augmented data or early stopping.

17. What is dropout and how does it help training?

    Dropout randomly deactivates a fraction of neurons during each training iteration, preventing co-adaptation and forcing redundancy. It acts as regularization to reduce overfitting.

18. What is batch normalization and why is it useful?

    Batch normalization normalizes layer inputs (zero mean, unit variance) per mini-batch. It stabilizes and speeds up training, allows higher learning rates, and provides slight regularization.

19. What is an epoch in neural network training?

    One complete pass through the entire training dataset. Training typically runs for many epochs.

20. What is a loss (cost) function and name one used for classification.

    A loss function measures the difference between predicted and actual outputs, guiding optimization. Cross-entropy loss is commonly used for classification; mean squared error (MSE) is used for regression.

21. What is the Adam optimizer?

    Adam (Adaptive Moment Estimation) is an optimization algorithm that combines momentum and adaptive per-parameter learning rates using estimates of first and second moments of gradients. It is widely used for fast, robust convergence.

See more Deep Learning and Natural Language Processing flashcards →

Planning Deep Learning and Natural Language Processing for Data Science & Machine Learning

Deep Learning and Natural Language Processing (Data Science & Machine Learning) FAQ