Deep Learning Frameworks and Production Syllabus

In Keras, what is the difference between saving in the .keras (or SavedModel) format versus saving only weights?

Saving the full model (`.keras`/SavedModel) stores architecture, weights, optimizer state, and training config, so it can be reloaded and used immediately. Saving only weights stores parameter values; you must first reconstruct an identical architecture before loading them.

What is data parallelism in distributed training?

The model is replicated on each device; each replica processes a different shard of the mini-batch, computes gradients locally, and gradients are synchronized (usually via all-reduce) and averaged so all replicas stay identical. It scales throughput with batch size.

What is model parallelism and when is it needed?

Model parallelism splits a single model's layers or tensors across multiple devices because the model is too large to fit in one device's memory. Different devices compute different parts of the model for the same batch. Used for very large models (e.g., huge LLMs).

What does the all-reduce operation do in distributed data-parallel training?

All-reduce aggregates (sums/averages) gradient tensors across all workers and distributes the identical reduced result back to every worker, keeping model replicas synchronized. Ring all-reduce achieves this with bandwidth roughly independent of the number of workers.

Compare PyTorch's DataParallel (DP) and DistributedDataParallel (DDP).

`DataParallel` uses a single process with multiple threads and a single GPU as parameter server (slower, GIL-bound). `DistributedDataParallel` uses one process per GPU with efficient all-reduce, scales across multiple nodes, and is the recommended approach for performance.

What is mixed precision training and what numeric formats does it combine?

Mixed precision performs most computations in a lower-precision float (FP16 or BF16) while keeping a master copy of weights and certain reductions in FP32. This reduces memory usage and speeds up training on hardware with tensor cores, with minimal accuracy loss.

What is loss scaling in FP16 mixed precision training and why is it needed?

FP16 has a limited dynamic range, so small gradient values can underflow to zero. Loss scaling multiplies the loss by a large factor before backward (shifting gradients into FP16's representable range), then unscales gradients before the optimizer step. Dynamic loss scaling adjusts the factor automatically.

How does BF16 differ from FP16, and why is BF16 often preferred for training?

BF16 (bfloat16) uses 8 exponent bits (same range as FP32) but only 7 mantissa bits, while FP16 uses 5 exponent and 10 mantissa bits. BF16's wider dynamic range avoids overflow/underflow, so it typically needs no loss scaling, at the cost of lower precision than FP16.

What is gradient accumulation and how does it produce a larger effective batch size?

Gradient accumulation runs the forward/backward pass over several micro-batches, summing gradients into `.grad` without stepping, then performs one optimizer step after $N$ micro-batches. The effective batch size is $N \times (\text{micro-batch size})$, enabling large-batch training within limited memory.

When using gradient accumulation, why might you divide the loss by the number of accumulation steps?

Because gradients are summed across the $N$ micro-batches. Dividing each micro-batch loss by $N$ (i.e., $\frac{1}{N}\sum L_i$) makes the accumulated gradient equal the mean gradient over the effective batch, matching a single large-batch update.

What is gradient (activation) checkpointing and what trade-off does it make?

Gradient checkpointing saves memory by NOT storing all intermediate activations during the forward pass; instead it recomputes them during the backward pass. The trade-off is reduced memory (allowing larger models/batches) at the cost of extra compute (roughly one additional forward pass).

How do GPUs and TPUs differ architecturally for deep learning?

GPUs are general-purpose massively parallel processors with many CUDA/tensor cores, flexible across frameworks. TPUs are Google's application-specific integrated circuits (ASICs) built around large systolic-array matrix-multiply units (MXUs), optimized for dense matrix operations and large-batch training, primarily via XLA/TensorFlow/JAX.

What is a systolic array and why is it central to TPU performance?

A systolic array is a grid of processing elements that pass data rhythmically to neighbors, performing many multiply-accumulate operations while minimizing memory reads/writes. It makes large matrix multiplications (the core of neural nets) highly efficient, which is why it underlies the TPU's MXU.

What is quantization in the context of neural network deployment?

Quantization reduces the numeric precision of weights and/or activations, typically from 32-bit float to lower-bit integers (e.g., INT8). This shrinks model size, lowers memory bandwidth, and speeds up inference on integer hardware, usually with a small accuracy drop.

Contrast post-training quantization (PTQ) with quantization-aware training (QAT).

PTQ quantizes an already-trained model without retraining (fast, but larger accuracy loss for aggressive bit-widths). QAT simulates quantization effects during training (inserting fake-quant nodes) so the model learns to be robust to reduced precision, generally yielding higher accuracy at the same bit-width.

Give the standard affine quantization formula mapping a real value to an integer.

$q = \text{round}\left(\frac{r}{s}\right) + z$, where $r$ is the real value, $s$ is the scale, $z$ is the zero-point, and $q$ is the quantized integer. Dequantization recovers $r \approx s\,(q - z)$.

What is pruning in neural networks, and how do structured and unstructured pruning differ?

Pruning removes weights (or larger components) deemed unimportant (often small-magnitude) to reduce size/compute. Unstructured pruning zeros individual weights, producing sparse matrices that need special hardware/libraries for speedup. Structured pruning removes whole units/channels/filters, giving dense smaller models that run faster on standard hardware.

What is knowledge distillation and what are the teacher and student?

Knowledge distillation trains a small 'student' model to mimic a large 'teacher' (or ensemble). The student learns from the teacher's soft outputs (probability distributions) in addition to hard labels, transferring 'dark knowledge' and often achieving accuracy close to the teacher at much lower cost.

In knowledge distillation, what is the role of the temperature $T$ in the softmax?

Temperature softens the probability distribution: $p_i = \frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)}$. A higher $T$ produces softer probabilities that reveal the relative similarities between classes, giving the student richer gradient signal. Gradients from the soft targets are often scaled by $T^{2}$.

What is ONNX and what problem does it solve?

ONNX (Open Neural Network Exchange) is an open, framework-agnostic standard format for representing machine learning models as a computation graph. It enables interoperability, so a model trained in one framework (e.g., PyTorch) can be exported and run in another runtime or hardware target (e.g., ONNX Runtime, TensorRT).

What is NVIDIA TensorRT and what optimizations does it perform?

TensorRT is NVIDIA's high-performance deep learning inference optimizer and runtime for GPUs. It optimizes models via layer/tensor fusion, precision calibration (FP16/INT8), kernel auto-tuning, and dynamic tensor memory management to minimize latency and maximize throughput at inference.

Name three common techniques for optimizing inference serving throughput and latency.

1) Dynamic/continuous batching (grouping requests to use hardware efficiently); 2) Reduced-precision inference (FP16/INT8 quantization); 3) Operator fusion / compiled graphs (e.g., TensorRT, ONNX Runtime). Others include model caching, KV-caching for LLMs, and hardware-specific kernels.

What is dynamic batching in model serving and what trade-off does it involve?

Dynamic batching collects individual inference requests arriving within a short time window into a single batch to improve GPU utilization and throughput. The trade-off is added per-request latency from waiting to fill the batch, tuned via a maximum batch size and timeout.

What is the purpose of experiment tracking tools like Weights & Biases and MLflow?

They log and organize ML experiments: hyperparameters, metrics, code/data versions, artifacts (models, plots), and system stats. This provides reproducibility, comparison across runs, visualization dashboards, and collaboration, so results are not lost in ad-hoc notebooks.

What are the four main components of MLflow?

1) MLflow Tracking (logging params, metrics, artifacts); 2) MLflow Projects (packaging code for reproducible runs); 3) MLflow Models (a standard packaging format for deployment); 4) MLflow Model Registry (versioning, staging, and lifecycle management of models).

What is hyperparameter tuning, and how do grid search and random search differ?

Hyperparameter tuning searches for the configuration (e.g., learning rate, depth) that optimizes validation performance. Grid search exhaustively evaluates all combinations on a predefined grid. Random search samples configurations randomly and often finds good values faster in high-dimensional spaces because not all hyperparameters matter equally.

How does Bayesian optimization improve hyperparameter search over grid/random search?

Bayesian optimization builds a probabilistic surrogate model (e.g., Gaussian process) of the objective and uses an acquisition function to choose the next configuration that balances exploration and exploitation. It uses information from past evaluations to converge to good hyperparameters in far fewer trials.

What is the Hyperband / successive halving strategy for hyperparameter tuning?

Successive halving allocates a small budget (e.g., epochs) to many configurations, keeps the top fraction, and repeatedly increases budget while pruning poor performers. Hyperband runs successive halving with multiple budget/bracket settings to balance the number of configs against per-config resources, efficiently killing bad trials early.

What factors must be controlled to make deep learning training runs reproducible?

Fixing all random seeds (Python, NumPy, framework, CUDA), pinning library and framework versions, versioning data and code, controlling hardware/nondeterministic GPU ops (e.g., `torch.use_deterministic_algorithms(True)`, cuDNN deterministic flags), and logging the exact configuration. Full bitwise reproducibility across different hardware is often impossible.

What is model versioning and why is it important in an ML workflow?

Model versioning assigns identifiable versions to trained models along with their associated code, data, hyperparameters, and metrics. It enables rollback to previous models, auditability, A/B comparison, reproducibility, and reliable promotion of models through staging to production (e.g., via an MLflow Model Registry).