Python TensorFlow Syllabus

What does a Dense (fully connected) layer compute?

It computes $y = \sigma(Wx + b)$, where $W$ is the weight matrix, $b$ the bias vector, $x$ the input, and $\sigma$ an activation function applied element-wise.

What is the ReLU activation function?

The Rectified Linear Unit is $f(x) = \max(0, x)$. It outputs the input if positive and 0 otherwise, introducing non-linearity while being cheap to compute and helping mitigate the vanishing gradient problem.

What is the softmax function and what is it used for?

Softmax converts a vector of logits into a probability distribution: $\sigma(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$. It is used in the output layer for multi-class classification so outputs sum to 1.

What is the sigmoid activation function?

The sigmoid (logistic) function is $\sigma(x) = \frac{1}{1 + e^{-x}}$, mapping any real number to $(0,1)$. It is commonly used for binary classification output and as a gating function.

What method trains a compiled Keras model, and what key arguments does it take?

model.fit(x, y, epochs=N, batch_size=B, validation_data=...). epochs sets how many full passes over the data, batch_size sets samples per gradient update, and validation_data monitors generalization each epoch.

Define an epoch, a batch, and an iteration in training.

An epoch is one full pass through the entire training dataset. A batch is a subset of samples processed in one forward/backward pass. An iteration (step) is one parameter update; iterations per epoch = ceil(dataset size / batch size).

What is gradient descent and its parameter update rule?

Gradient descent minimizes a loss by stepping parameters opposite the gradient: $\theta \leftarrow \theta - \eta \nabla_{\theta} J(\theta)$, where $\eta$ is the learning rate and $\nabla_{\theta} J$ is the gradient of the loss with respect to the parameters.

What is the role of the learning rate $\eta$?

The learning rate controls the step size of each parameter update. Too high $\eta$ causes divergence or oscillation; too low $\eta$ causes slow convergence and risk of getting stuck. It is a critical hyperparameter.

What is the Adam optimizer?

Adam (Adaptive Moment Estimation) is an optimizer that maintains per-parameter adaptive learning rates using running estimates of the first moment (mean) and second moment (uncentered variance) of the gradients, combining the benefits of momentum and RMSProp.

Give the formula for Mean Squared Error (MSE) loss.

$$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$ where $y_i$ is the true value and $\hat{y}_i$ the prediction. It is the standard loss for regression.

Give the formula for binary cross-entropy loss.

$$L = -\frac{1}{n}\sum_{i=1}^{n}\left[y_i \log(\hat{y}_i) + (1-y_i)\log(1-\hat{y}_i)\right]$$ used for binary classification, where $\hat{y}_i$ is the predicted probability and $y_i \in \{0,1\}$.

What is tf.GradientTape used for?

tf.GradientTape records operations during a forward pass onto a 'tape' so that automatic differentiation can compute gradients of a target (loss) with respect to watched variables via tape.gradient(loss, variables). It is the core of custom training loops.

What are the essential steps inside a custom training loop in TensorFlow 2?

1) Open a tf.GradientTape context and compute predictions and loss; 2) compute gradients = tape.gradient(loss, model.trainable_variables); 3) apply them with optimizer.apply_gradients(zip(gradients, model.trainable_variables)). Repeat per batch/epoch.

What does optimizer.apply_gradients() do?

It updates the model's trainable variables using the computed gradients according to the optimizer's update rule. You pass it pairs of (gradient, variable), typically as zip(grads, model.trainable_variables).

What does model.evaluate() return and do?

model.evaluate(x_test, y_test) runs the model on test data and returns the loss and the metric values (e.g., accuracy) specified at compile time. It performs no weight updates; it only measures performance.

What is the difference between model.predict() and model.evaluate()?

model.predict() returns the model's raw output predictions for given inputs (no labels needed). model.evaluate() requires labels and returns aggregate loss/metrics measuring how well predictions match the true labels.

Define accuracy, precision, and recall as classification metrics.

Accuracy $= \frac{TP+TN}{TP+TN+FP+FN}$. Precision $= \frac{TP}{TP+FP}$ (correctness of positive predictions). Recall $= \frac{TP}{TP+FN}$ (coverage of actual positives).

What is the F1 score and its formula?

The F1 score is the harmonic mean of precision and recall: $$F_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}$$ It balances the two and is useful for imbalanced classes.

What is distributed training and what API does TensorFlow provide for it?

Distributed training spreads computation across multiple GPUs/TPUs or machines to speed up training. TensorFlow provides the tf.distribute.Strategy API to do this with minimal code changes.

What does tf.distribute.MirroredStrategy do?

MirroredStrategy implements synchronous data-parallel training across multiple GPUs on a single machine. It replicates all model variables (mirrors them) on each device, and aggregates gradients with all-reduce so replicas stay in sync.

Compare MirroredStrategy, MultiWorkerMirroredStrategy, and TPUStrategy.

MirroredStrategy: synchronous training across multiple GPUs on one machine. MultiWorkerMirroredStrategy: synchronous training across multiple machines (workers). TPUStrategy: synchronous training on Cloud TPUs / TPU pods.

What is the difference between data parallelism and model parallelism?

Data parallelism replicates the whole model on each device and splits the data batch across them, syncing gradients. Model parallelism splits a single (too-large) model across devices, with each device computing part of the network.

What is TensorFlow Serving?

TensorFlow Serving is a high-performance, production-grade serving system for deploying trained ML models as scalable network services. It supports model versioning, hot-swapping new versions, and exposes gRPC and REST APIs for inference.

What model format does TensorFlow Serving consume, and how is it produced?

It consumes the SavedModel format (a directory with saved_model.pb plus a variables/ and assets/ folders). You produce it with model.save('path') or tf.saved_model.save(model, 'path').

What is TensorFlow Lite (TFLite)?

TensorFlow Lite is a lightweight framework for deploying models on mobile, embedded, and IoT/edge devices. It uses an optimized .tflite FlatBuffer format and a small interpreter designed for low latency and small binary size.

How do you convert a model to TensorFlow Lite, and what optimization reduces its size?

Use tf.lite.TFLiteConverter (e.g., from_saved_model or from_keras_model) and call convert(). Quantization (e.g., post-training quantization to int8) reduces model size and speeds up inference with minimal accuracy loss.

What is TensorFlow Extended (TFX)?

TFX is an end-to-end platform for deploying production ML pipelines. It chains components like ExampleGen, StatisticsGen, SchemaGen, Transform, Trainer, Evaluator, and Pusher to handle data validation, training, evaluation, and deployment at scale.

What is TensorFlow Hub?

TensorFlow Hub is a repository of reusable, pre-trained model modules and components. It enables transfer learning: you can load a pretrained module (e.g., via hub.KerasLayer) and fine-tune it on your own task instead of training from scratch.

What is TensorFlow.js?

TensorFlow.js is a JavaScript library for training and running ML models directly in the browser or in Node.js, using WebGL/WebGPU for GPU acceleration. It can also import and run existing Python-trained TensorFlow/Keras models in the browser.

What is a Convolutional Neural Network (CNN) and why is it used for image classification?

A CNN is a neural network using convolutional layers that apply learnable filters/kernels over local regions of an image to detect spatial features (edges, textures, shapes). Parameter sharing and local connectivity make it efficient and translation-aware, making it the standard for image classification. In Keras: tf.keras.layers.Conv2D and MaxPooling2D.

What are word embeddings in TensorFlow NLP, and which layer creates them?

Word embeddings are dense, low-dimensional vector representations of words that capture semantic similarity (similar words map to nearby vectors). In Keras they are learned via tf.keras.layers.Embedding, which maps integer token indices to trainable vectors. RNN/LSTM or Transformer layers then process these sequences.