Deep Learning Frameworks and Production Syllabus

Frameworks and Production syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Frameworks and Production in Deep Learning, not a summary of it.

1. 1

   PyTorch

   4 topics

   • Tensors and Autograd
   • nn.Module and Model Building
   • DataLoaders and Datasets
   • Training Loops and torch.compile
2. 2

   TensorFlow and Keras

   3 topics

   • Keras Sequential and Functional API
   • Eager Execution and tf.function
   • Model Saving and SavedModel Format
3. 3

   Training at Scale

   4 topics

   • Distributed Training
   • Mixed Precision Training
   • Gradient Accumulation and Checkpointing
   • GPU and TPU Hardware
4. 4

   Model Optimization and Deployment

   4 topics

   • Quantization
   • Pruning and Knowledge Distillation
   • ONNX and TensorRT
   • Serving and Inference Optimization
5. 5

   MLOps and Experimentation

   4 topics

   • Experiment Tracking (Weights & Biases, MLflow)
   • Hyperparameter Tuning
   • Model Versioning and Reproducibility
   • Monitoring and Drift Detection

Frameworks and Production flashcards for Deep Learning

18 of 50 cards from the Frameworks and Production deck — real questions with worked answers.

1. In PyTorch, what is a tensor and how does it differ from a NumPy array?

   A tensor is a multi-dimensional array (the core data structure in PyTorch) that supports GPU acceleration and automatic differentiation. Unlike a NumPy array, it can live on a GPU (via `.to('cuda')`) and can track gradients when `requires_grad=True`.

2. What does PyTorch's autograd engine do, and how is a gradient computed for a scalar loss?

   Autograd records operations on tensors with `requires_grad=True` into a dynamic computation graph, then performs reverse-mode automatic differentiation. Calling `loss.backward()` computes $\frac{\partial \text{loss}}{\partial \theta}$ for every leaf parameter and accumulates it into each parameter's `.grad` attribute.

3. Why must you call optimizer.zero_grad() in a PyTorch training loop, and what happens if you don't?

   Because `.backward()` accumulates (adds) gradients into `.grad` rather than overwriting them. Without `zero_grad()`, gradients from previous iterations sum together, giving incorrect updates. The exception is deliberate gradient accumulation across micro-batches.

4. What is the purpose of torch.no_grad() and when is it used?

   `torch.no_grad()` is a context manager that disables gradient tracking, so operations are not recorded in the autograd graph. It reduces memory and speeds up inference/evaluation and manual parameter updates where backpropagation is not needed.

5. In PyTorch, what is the difference between a leaf tensor and a non-leaf tensor with respect to gradients?

   A leaf tensor is one created directly by the user (e.g., model parameters) with no history; its `.grad` is populated after `backward()`. A non-leaf tensor results from an operation; by default its gradient is not retained unless `.retain_grad()` is called.

6. What is nn.Module in PyTorch and which two methods define a custom model?

   `nn.Module` is the base class for all neural network components. A custom model subclasses it, defines layers/parameters in `__init__` (calling `super().__init__()`), and implements the `forward(self, x)` method describing the computation. Autograd handles the backward pass automatically.

7. In PyTorch, why do you call model(x) instead of model.forward(x) directly?

   Calling the module instance invokes `__call__`, which runs registered forward and backward hooks and other bookkeeping before dispatching to `forward()`. Calling `forward()` directly bypasses these hooks.

8. What is the difference between model.parameters() and model.buffers() in PyTorch?

   `parameters()` returns learnable tensors (registered via `nn.Parameter`) that the optimizer updates and autograd tracks. `buffers()` returns non-learnable persistent state (e.g., BatchNorm running mean/variance) that is saved in the state_dict but not optimized.

9. What do model.train() and model.eval() control in PyTorch?

   They set the module's training flag, changing the behavior of layers like Dropout (active in train, disabled in eval) and BatchNorm (uses batch statistics in train, running statistics in eval). They do NOT enable/disable gradient computation.

10. What roles do Dataset and DataLoader play in PyTorch?

    A `Dataset` defines `__len__` and `__getitem__` to provide individual samples. A `DataLoader` wraps a Dataset to handle batching, shuffling, parallel loading via `num_workers`, and collation, yielding mini-batches during training.

11. What is the difference between a map-style and an iterable-style Dataset in PyTorch?

    A map-style dataset implements `__getitem__` and `__len__`, allowing random access by index (good for shuffling). An iterable-style dataset (subclass of `IterableDataset`) implements `__iter__` and is used for streaming data where the length or random access is unavailable.

12. What are the standard five steps inside a PyTorch training loop iteration?

    1) `optimizer.zero_grad()`; 2) forward pass `outputs = model(x)`; 3) compute `loss = criterion(outputs, y)`; 4) `loss.backward()` to compute gradients; 5) `optimizer.step()` to update parameters.

13. What does torch.compile do and what is its main benefit?

    Introduced in PyTorch 2.0, `torch.compile(model)` JIT-compiles the model using TorchDynamo (graph capture) and a backend like TorchInductor to fuse operations and generate optimized kernels, typically speeding up training/inference while keeping eager-mode ease of use.

14. Compare the Keras Sequential API and Functional API.

    The Sequential API stacks layers in a simple linear pipeline (one input, one output, no branching). The Functional API treats layers as callables on tensors, enabling arbitrary directed acyclic graphs: multiple inputs/outputs, shared layers, skip connections, and branching.

15. Which types of model architectures cannot be built with the Keras Sequential API?

    Models with multiple inputs or outputs, shared layers, residual/skip connections, or any non-linear (branching, merging) topology. These require the Functional API or model subclassing.

16. What is eager execution in TensorFlow?

    Eager execution (default since TF 2.x) evaluates operations immediately as they are called, returning concrete values rather than building a static graph first. It makes debugging intuitive and Pythonic, at some cost to performance versus graph mode.

17. What does the @tf.function decorator do in TensorFlow?

    It traces a Python function and compiles it into a static, optimized TensorFlow graph (via AutoGraph). This enables graph-level optimizations, faster execution, and portability, combining eager's ease of writing with graph mode's performance.

18. What is 'retracing' in tf.function and why can it hurt performance?

    Retracing is when tf.function builds a new graph because it is called with a new input signature (different shapes/dtypes or Python argument values). Excessive retracing is slow and wasteful; it is mitigated by using consistent tensor shapes and `input_signature` or `tf.TensorSpec`.

See more Frameworks and Production flashcards →

Planning Frameworks and Production for Deep Learning

Frameworks and Production (Deep Learning) FAQ