Deep Learning Foundations of Neural Networks Syllabus

Foundations of Neural Networks syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Foundations of Neural Networks in Deep Learning, not a summary of it.

1. 1

   Machine Learning Prerequisites

   4 topics

   • Supervised vs Unsupervised Learning
   • Bias-Variance Tradeoff
   • Loss Functions and Cost Surfaces
   • Train, Validation, and Test Splits
2. 2

   Mathematical Foundations

   4 topics

   • Linear Algebra for Deep Learning
   • Calculus and Gradients
   • Probability and Statistics
   • Information Theory Basics
3. 3

   The Perceptron and MLPs

   4 topics

   • Biological and Artificial Neurons
   • Multilayer Perceptrons
   • Activation Functions
   • Universal Approximation Theorem
4. 4

   Backpropagation and Training

   4 topics

   • Forward and Backward Pass
   • Gradient Descent Variants
   • Optimizers
   • Vanishing and Exploding Gradients
5. 5

   Regularization and Generalization

   4 topics

   • L1 and L2 Regularization
   • Dropout
   • Normalization Techniques
   • Early Stopping and Data Augmentation

Foundations of Neural Networks flashcards for Deep Learning

21 of 50 cards from the Foundations of Neural Networks deck — real questions with worked answers.

1. What distinguishes supervised from unsupervised learning?

   Supervised learning trains on labeled data $\{(x_i, y_i)\}$ to learn a mapping $f: x \to y$ (classification/regression). Unsupervised learning uses unlabeled data $\{x_i\}$ to discover structure (clustering, dimensionality reduction, density estimation).

2. Define the bias-variance tradeoff and give the expected test error decomposition.

   Expected error decomposes as $$\mathbb{E}[(y-\hat{f}(x))^{2}] = \underbrace{(\text{Bias}[\hat{f}])^{2}}_{\text{underfit}} + \underbrace{\text{Var}[\hat{f}]}_{\text{overfit}} + \sigma^{2}_{\text{irreducible}}.$$ Increasing model complexity lowers bias but raises variance.

3. What are high bias and high variance symptoms in terms of training/test error?

   High bias (underfitting): high training error and high test error. High variance (overfitting): low training error but high test error (large gap between them).

4. Write the mean squared error (MSE) loss for a regression problem.

   $$\text{MSE} = \frac{1}{N}\sum_{i=1}^{N}(y_i - \hat{y}_i)^{2}$$

5. Write the binary cross-entropy loss for a single example with label $y\in\{0,1\}$ and prediction $\hat{y}$.

   $$L = -\big[y\log(\hat{y}) + (1-y)\log(1-\hat{y})\big]$$

6. Write the categorical cross-entropy loss for $C$ classes with one-hot target $y$ and softmax output $\hat{y}$.

   $$L = -\sum_{c=1}^{C} y_c \log(\hat{y}_c)$$

7. What is a cost/loss surface, and why can it be problematic for optimization?

   The cost surface is the graph of loss as a function of the model parameters. In deep networks it is high-dimensional and non-convex, containing local minima, saddle points, plateaus, and sharp/flat regions that complicate gradient-based optimization.

8. What are the roles of the training, validation, and test splits?

   Training set fits model parameters; validation set tunes hyperparameters and enables model selection/early stopping; test set gives an unbiased final estimate of generalization and must not influence any modeling decision.

9. What is $k$-fold cross-validation?

   Data is split into $k$ folds; the model is trained on $k-1$ folds and validated on the held-out fold, rotating so each fold is validation once. The performance estimate is the average over the $k$ runs, reducing variance of the estimate.

10. Why should the test set never be used during training or hyperparameter tuning?

    Using it would leak information and cause optimistic bias, so the reported performance would no longer be an unbiased estimate of true generalization to unseen data.

11. What is the dot product of two vectors $\vec{a},\vec{b}\in\mathbb{R}^{n}$, and what does it measure?

    $$\vec{a}\cdot\vec{b} = \sum_{i=1}^{n} a_i b_i = \|\vec{a}\|\,\|\vec{b}\|\cos\theta$$ It measures alignment/projection; it is $0$ when the vectors are orthogonal.

12. For matrices $A\in\mathbb{R}^{m\times n}$ and $B\in\mathbb{R}^{n\times p}$, what is the shape and entry formula of $C=AB$?

    $C\in\mathbb{R}^{m\times p}$ with entries $$C_{ij} = \sum_{k=1}^{n} A_{ik}B_{kj}.$$ Inner dimensions must match ($n$).

13. Define an eigenvector and eigenvalue of a square matrix $A$.

    A nonzero vector $\vec{v}$ is an eigenvector with eigenvalue $\lambda$ if $$A\vec{v} = \lambda\vec{v},$$ meaning $A$ only scales $\vec{v}$ without changing its direction.

14. Give the formulas for the $L_1$ and $L_2$ (Euclidean) norms of a vector $\vec{x}$.

    $$\|\vec{x}\|_1 = \sum_i |x_i|, \qquad \|\vec{x}\|_2 = \sqrt{\sum_i x_i^{2}}$$

15. What is the gradient $\nabla f$ of a scalar function $f(\vec{x})$, and what direction does it point?

    $$\nabla f = \left(\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}\right)$$ It points in the direction of steepest ascent; $-\nabla f$ is steepest descent.

16. State the chain rule for a composition $f(g(x))$.

    $$\frac{d}{dx}f(g(x)) = f'(g(x))\cdot g'(x)$$ For deep nets this generalizes to multiplying Jacobians layer by layer.

17. What is a partial derivative $\frac{\partial f}{\partial x_i}$?

    The rate of change of $f$ with respect to $x_i$ while holding all other variables constant. It is the $i$-th component of the gradient.

18. State Bayes' theorem.

    $$P(A\mid B) = \frac{P(B\mid A)\,P(A)}{P(B)}$$ where $P(A)$ is the prior, $P(B\mid A)$ the likelihood, and $P(A\mid B)$ the posterior.

19. Give the formulas for the expectation and variance of a random variable $X$.

    $$\mathbb{E}[X] = \sum_x x\,P(x)\;\;(\text{or }\int x\,p(x)\,dx), \qquad \text{Var}(X) = \mathbb{E}[X^{2}] - (\mathbb{E}[X])^{2}$$

20. Write the probability density function of a Gaussian (normal) distribution.

    $$p(x) = \frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\!\left(-\frac{(x-\mu)^{2}}{2\sigma^{2}}\right)$$

21. Define Shannon entropy $H(X)$ of a discrete distribution.

    $$H(X) = -\sum_{i} p(x_i)\log p(x_i)$$ It measures the average uncertainty (in bits if $\log_2$) of the distribution.

See more Foundations of Neural Networks flashcards →

Planning Foundations of Neural Networks for Deep Learning

Foundations of Neural Networks (Deep Learning) FAQ