Deep Learning Foundations of Neural Networks Syllabus

Define Shannon entropy $H(X)$ of a discrete distribution.

$$H(X) = -\sum_{i} p(x_i)\log p(x_i)$$ It measures the average uncertainty (in bits if $\log_2$) of the distribution.

Define the Kullback-Leibler (KL) divergence between distributions $P$ and $Q$.

$$D_{KL}(P\parallel Q) = \sum_i p(i)\log\frac{p(i)}{q(i)}$$ It is $\geq 0$, equals $0$ iff $P=Q$, and is asymmetric ($D_{KL}(P\parallel Q)\neq D_{KL}(Q\parallel P)$).

How is cross-entropy related to entropy and KL divergence?

$$H(P,Q) = H(P) + D_{KL}(P\parallel Q) = -\sum_i p(i)\log q(i)$$ Minimizing cross-entropy over $Q$ (fixed $P$) is equivalent to minimizing $D_{KL}(P\parallel Q)$.

Compare a biological neuron to an artificial neuron.

Biological: dendrites receive signals, the soma integrates them, and an axon fires a spike past a threshold. Artificial: inputs are multiplied by weights, summed with a bias, and passed through an activation function that determines the output.

Write the computation performed by a single artificial neuron.

$$y = \phi\!\left(\sum_{i=1}^{n} w_i x_i + b\right) = \phi(\vec{w}^{\top}\vec{x} + b)$$ where $\phi$ is the activation function and $b$ the bias.

What is a perceptron and what is its key limitation?

A perceptron is a single neuron with a step activation, computing a linear decision boundary. Its limitation: it can only separate linearly separable data and cannot solve XOR.

What is a Multilayer Perceptron (MLP)?

A feedforward neural network with one or more hidden layers of neurons using nonlinear activations, fully connected between layers, capable of learning nonlinear decision boundaries.

Why are nonlinear activation functions necessary in an MLP?

Without nonlinearity, stacking linear layers collapses to a single linear transformation $W_{\text{eff}}\vec{x}+b$, so the network could only represent linear functions regardless of depth.

Write the sigmoid function and its derivative.

$$\sigma(x) = \frac{1}{1+e^{-x}}, \qquad \sigma'(x) = \sigma(x)\big(1-\sigma(x)\big)$$ Output range $(0,1)$.

Write the tanh function and its output range.

$$\tanh(x) = \frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$$ Zero-centered with output range $(-1,1)$.

Write the ReLU activation and its derivative.

$$\text{ReLU}(x) = \max(0,x), \qquad \frac{d}{dx}\text{ReLU}(x)=\begin{cases}1 & x>0\\ 0 & x<0\end{cases}$$

Write the Leaky ReLU function and state the problem it addresses.

$$\text{LeakyReLU}(x)=\begin{cases}x & x>0\\ \alpha x & x\leq 0\end{cases}\quad (\text{small }\alpha)$$ It fixes the 'dying ReLU' problem by giving a nonzero gradient for negative inputs.

Write the softmax function for logits $z_1,\dots,z_C$.

$$\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{C} e^{z_j}}$$ It outputs a probability distribution over $C$ classes summing to $1$.

State the Universal Approximation Theorem.

A feedforward network with a single hidden layer containing a finite number of neurons and a suitable non-constant, bounded (nonlinear) activation can approximate any continuous function on a compact domain to arbitrary accuracy.

What does the Universal Approximation Theorem NOT guarantee?

It guarantees existence of approximating weights but says nothing about how many neurons are needed, whether training can find those weights, or that the solution will generalize. Depth can make representation exponentially more efficient than width.

Describe the forward pass in a neural network.

Inputs propagate layer by layer: each layer computes $\vec{z}^{(l)} = W^{(l)}\vec{a}^{(l-1)} + \vec{b}^{(l)}$ and $\vec{a}^{(l)} = \phi(\vec{z}^{(l)})$, until the output layer produces predictions used to evaluate the loss.

What is the purpose of the backward pass (backpropagation)?

It computes the gradient of the loss with respect to every parameter by applying the chain rule backward from the output to the input, propagating error signals $\delta^{(l)}=\frac{\partial L}{\partial \vec{z}^{(l)}}$ so parameters can be updated.

Write the backpropagation error recursion for hidden layer $l$.

$$\delta^{(l)} = \big(W^{(l+1)\top}\delta^{(l+1)}\big)\odot \phi'(\vec{z}^{(l)})$$ where $\odot$ is elementwise multiplication.

Write the gradient descent parameter update rule.

$$\theta \leftarrow \theta - \eta\,\nabla_\theta L(\theta)$$ where $\eta$ is the learning rate.

Compare batch, stochastic (SGD), and mini-batch gradient descent.

Batch GD uses the whole dataset per update (stable but slow). SGD uses one example per update (fast, noisy). Mini-batch uses a small subset (e.g. 32-256), balancing speed, noise, and hardware efficiency.

What is momentum in gradient descent and what problem does it solve?

$$v_t = \gamma v_{t-1} + \eta\nabla_\theta L,\qquad \theta \leftarrow \theta - v_t$$ It accumulates a velocity to accelerate along consistent directions and dampen oscillations in ravines, speeding convergence.

Describe the RMSProp optimizer update.

It keeps a decaying average of squared gradients and scales the step per parameter: $$s_t=\beta s_{t-1}+(1-\beta)g_t^{2},\quad \theta\leftarrow\theta-\frac{\eta}{\sqrt{s_t}+\epsilon}g_t.$$ Adapts the learning rate to each parameter.

Write the Adam optimizer update rule (with bias correction).

$$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^{2}$$ $$\hat{m}_t=\frac{m_t}{1-\beta_1^{t}},\ \hat{v}_t=\frac{v_t}{1-\beta_2^{t}},\quad \theta\leftarrow\theta-\frac{\eta}{\sqrt{\hat{v}_t}+\epsilon}\hat{m}_t$$

What roles do $\beta_1$ and $\beta_2$ play in Adam, and their typical values?

$\beta_1$ (typically $0.9$) controls the decay of the first-moment (mean) estimate; $\beta_2$ (typically $0.999$) controls the second-moment (uncentered variance) estimate. $\epsilon\approx 10^{-8}$ prevents division by zero.

What is the vanishing gradient problem?

In deep networks, repeated multiplication of small derivatives (e.g. sigmoid/tanh, whose gradients are $<1$) during backprop shrinks gradients exponentially toward earlier layers, so early weights update extremely slowly or stop learning.

What is the exploding gradient problem and a common remedy?

Repeated multiplication of large values makes gradients grow exponentially, causing unstable, diverging updates (NaNs). A common remedy is gradient clipping, which rescales gradients whose norm exceeds a threshold.

List techniques that mitigate vanishing/exploding gradients.

ReLU-type activations, careful weight initialization (Xavier/He), batch normalization, residual/skip connections, gradient clipping, and gated architectures (LSTM/GRU) for sequences.

Write the L2-regularized (weight decay) objective and its effect.

$$L_{\text{total}} = L_{\text{data}} + \frac{\lambda}{2}\sum_j w_j^{2}$$ It penalizes large weights, shrinking them smoothly toward (but not exactly) zero to reduce overfitting.

Write the L1-regularized objective and contrast its effect with L2.

$$L_{\text{total}} = L_{\text{data}} + \lambda\sum_j |w_j|$$ L1 induces sparsity, driving some weights to exactly $0$ (feature selection); L2 shrinks weights smoothly but keeps them nonzero.

What is dropout and how does it behave differently during training vs inference?

During training, each neuron is kept with probability $p$ (dropped otherwise) at random, forcing redundant, robust features and acting like an ensemble. At inference all neurons are used, with activations scaled (e.g. by $p$, or inverted scaling during training) to match expected magnitudes.