Deep Learning Generative Deep Learning Syllabus

Generative Deep Learning syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Generative Deep Learning in Deep Learning, not a summary of it.

1. 1

   Autoencoders

   2 topics

   • Vanilla and Denoising Autoencoders
   • Variational Autoencoders
2. 2

   Generative Adversarial Networks

   3 topics

   • GAN Framework
   • Training Instability and Mode Collapse
   • DCGAN, StyleGAN, CycleGAN
3. 3

   Diffusion Models

   4 topics

   • Denoising Diffusion Probabilistic Models
   • Latent Diffusion and Stable Diffusion
   • Classifier-Free Guidance
   • Text-to-Image Generation
4. 4

   Autoregressive Generation

   2 topics

   • PixelRNN and PixelCNN
   • Sampling Strategies (Top-k, Top-p, Temperature)

Generative Deep Learning flashcards for Deep Learning

19 of 50 cards from the Generative Deep Learning deck — real questions with worked answers.

1. What is the core objective of a vanilla autoencoder?

   To learn a compressed latent representation by training an encoder $f$ and decoder $g$ to reconstruct the input, minimizing a reconstruction loss such as $\mathcal{L} = \lVert x - g(f(x)) \rVert^{2}$. The bottleneck layer forces the network to capture the most salient features.

2. How does a denoising autoencoder (DAE) differ from a vanilla autoencoder?

   A DAE corrupts the input to $\tilde{x}$ (e.g., adding Gaussian noise or masking) and trains the network to reconstruct the clean original: $\mathcal{L} = \lVert x - g(f(\tilde{x})) \rVert^{2}$. This prevents learning the identity function and yields more robust features.

3. Why is a plain autoencoder generally a poor generative model?

   Its latent space is not regularized to any known prior distribution, so it has 'holes' and discontinuities. Sampling arbitrary latent points and decoding them typically produces unrealistic outputs, so you cannot reliably generate new samples.

4. What two terms make up the Variational Autoencoder (VAE) loss (ELBO)?

   A reconstruction term and a KL-divergence regularizer: $$\mathcal{L} = \mathbb{E}_{q(z|x)}[\log p(x|z)] - D_{KL}\!\big(q(z|x)\,\Vert\,p(z)\big),$$ where the ELBO is maximized (or its negative minimized).

5. What prior distribution does a standard VAE assume over the latent variable $z$?

   An isotropic standard Gaussian, $p(z) = \mathcal{N}(0, I)$. The KL term pushes the approximate posterior $q(z|x)$ toward this prior.

6. What is the reparameterization trick in a VAE and why is it needed?

   It rewrites the sampled latent as $z = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$, moving the stochasticity to $\epsilon$. This makes $z$ a deterministic differentiable function of $\mu$ and $\sigma$, allowing gradients to backpropagate through the sampling step.

7. What does the encoder of a VAE output?

   The parameters of the approximate posterior $q(z|x)$ — typically the mean vector $\mu$ and log-variance $\log \sigma^{2}$ of a diagonal Gaussian — rather than a single deterministic code.

8. Write the closed-form KL divergence between $q(z|x)=\mathcal{N}(\mu,\sigma^{2})$ and $p(z)=\mathcal{N}(0,I)$.

   $$D_{KL} = -\tfrac{1}{2} \sum_{j=1}^{d} \big(1 + \log \sigma_{j}^{2} - \mu_{j}^{2} - \sigma_{j}^{2}\big).$$

9. What is 'posterior collapse' in VAEs?

   When the decoder becomes powerful enough to ignore $z$, driving $q(z|x)$ to match the prior $p(z)$ so the KL term vanishes and the latent code carries no information about the input. The model then reconstructs from the decoder alone.

10. What is the role of the $\beta$ hyperparameter in a $\beta$-VAE?

    It scales the KL term, $\mathcal{L} = \text{recon} + \beta \, D_{KL}$. Setting $\beta > 1$ increases pressure toward the prior, encouraging more disentangled latent factors at the cost of reconstruction fidelity.

11. Describe the two-player game in the GAN framework.

    A generator $G$ maps noise $z$ to fake samples, while a discriminator $D$ estimates the probability a sample is real. They play a minimax game: $G$ tries to fool $D$, and $D$ tries to distinguish real from generated data.

12. State the original GAN minimax objective.

    $$\min_{G} \max_{D} \; \mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_{z}}[\log (1 - D(G(z)))].$$

13. At the theoretical optimum of a GAN, what is the discriminator's output and the generator's distribution?

    The optimal discriminator is $D^{*}(x) = \dfrac{p_{data}(x)}{p_{data}(x) + p_{g}(x)} = \tfrac{1}{2}$ when $p_{g} = p_{data}$. At convergence the generator distribution equals the data distribution and $D$ outputs $\tfrac{1}{2}$ everywhere.

14. What is the 'non-saturating' generator loss and why is it used?

    Instead of minimizing $\log(1 - D(G(z)))$, the generator maximizes $\log D(G(z))$ (equivalently minimizes $-\log D(G(z))$). This provides stronger, non-vanishing gradients early in training when $D$ easily rejects fakes.

15. What divergence does the original GAN objective minimize between $p_{data}$ and $p_{g}$?

    The Jensen–Shannon divergence, $JSD(p_{data} \Vert p_{g})$, up to constants when the discriminator is optimal.

16. What is mode collapse in GAN training?

    When the generator produces only a small set of very similar outputs, ignoring the diversity of the data distribution. It maps many latent inputs to a few 'safe' modes that fool the current discriminator.

17. Name three common causes or symptoms of GAN training instability.

    Vanishing gradients (discriminator too strong), mode collapse (generator diversity loss), and oscillation/non-convergence where $G$ and $D$ chase each other without reaching equilibrium.

18. How does the Wasserstein GAN (WGAN) improve stability over the original GAN?

    It minimizes the Earth-Mover (Wasserstein-1) distance instead of JS divergence, giving smoother gradients even when distributions do not overlap. The critic outputs an unbounded score and must be Lipschitz-constrained (via weight clipping or a gradient penalty).

19. What is the gradient penalty in WGAN-GP and what does it enforce?

    A term $\lambda \, \mathbb{E}_{\hat{x}}\big[(\lVert \nabla_{\hat{x}} D(\hat{x}) \rVert_{2} - 1)^{2}\big]$ added to the critic loss, enforcing the 1-Lipschitz constraint by penalizing gradient norms that deviate from 1 on interpolated samples $\hat{x}$.

See more Generative Deep Learning flashcards →

Planning Generative Deep Learning for Deep Learning

Generative Deep Learning (Deep Learning) FAQ