Deep Learning Generative Deep Learning Syllabus

What is the central architectural innovation of StyleGAN?

A style-based generator: a mapping network transforms $z$ into an intermediate latent $w$, which controls each layer's feature statistics via Adaptive Instance Normalization (AdaIN). This disentangles high-level attributes and enables fine-grained, scale-specific style control.

What is AdaIN in StyleGAN and its formula?

Adaptive Instance Normalization injects style by normalizing each feature map and rescaling with style-derived parameters: $$\text{AdaIN}(x_{i}, y) = y_{s,i}\,\frac{x_{i} - \mu(x_{i})}{\sigma(x_{i})} + y_{b,i},$$ where $y_{s}$ and $y_{b}$ come from the style vector $w$.

What problem does CycleGAN solve and what is its key loss?

Unpaired image-to-image translation between two domains. Its key component is the cycle-consistency loss: translating $X \to Y \to X$ (and vice versa) should recover the original, $\lVert F(G(x)) - x \rVert_{1}$, ensuring content preservation without paired data.

Why does CycleGAN use two generators and two discriminators?

It learns mappings in both directions, $G: X \to Y$ and $F: Y \to X$, each paired with a discriminator ($D_{Y}$, $D_{X}$). The two adversarial losses plus the cycle-consistency loss enforce realistic and content-preserving bidirectional translation.

What are the two processes that define a Denoising Diffusion Probabilistic Model (DDPM)?

A fixed forward (diffusion) process that gradually adds Gaussian noise to data over $T$ steps until it becomes pure noise, and a learned reverse process that denoises step by step to generate samples.

Write the DDPM forward transition for a single step.

$$q(x_{t} \mid x_{t-1}) = \mathcal{N}\!\big(x_{t};\, \sqrt{1 - \beta_{t}}\, x_{t-1},\, \beta_{t} I\big),$$ where $\beta_{t}$ is the variance schedule at step $t$.

Give the closed-form for sampling $x_{t}$ directly from $x_{0}$ in DDPM.

$$x_{t} = \sqrt{\bar{\alpha}_{t}}\, x_{0} + \sqrt{1 - \bar{\alpha}_{t}}\, \epsilon, \quad \epsilon \sim \mathcal{N}(0, I),$$ where $\alpha_{t} = 1 - \beta_{t}$ and $\bar{\alpha}_{t} = \prod_{s=1}^{t} \alpha_{s}$.

What is the simplified DDPM training objective?

A noise-prediction MSE: $$\mathcal{L}_{simple} = \mathbb{E}_{t, x_{0}, \epsilon}\Big[\big\lVert \epsilon - \epsilon_{\theta}(x_{t}, t) \big\rVert^{2}\Big],$$ where the network $\epsilon_{\theta}$ predicts the noise added at step $t$.

What neural network architecture is typically used to parameterize the DDPM reverse process?

A U-Net with residual blocks, self-attention layers, and a sinusoidal time-step embedding, which predicts the noise $\epsilon_{\theta}(x_{t}, t)$ (or the denoised mean) at each diffusion step.

How does Latent Diffusion (LDM) reduce the cost of diffusion models?

It runs the diffusion process in a compressed latent space produced by a pretrained autoencoder (encoder/decoder), rather than on high-resolution pixels. This drastically cuts computation while preserving perceptual quality, since the latent space is lower-dimensional.

What are the main components of Stable Diffusion?

A VAE encoder/decoder (pixel↔latent), a U-Net denoiser operating in latent space, and a text encoder (e.g., a CLIP text model) whose embeddings condition the U-Net via cross-attention.

How is text conditioning injected into the Stable Diffusion U-Net?

Via cross-attention layers, where the text-embedding sequence provides the keys and values and the U-Net's spatial features provide the queries, letting the prompt guide denoising at each resolution.

What is classifier guidance in diffusion models?

Using the gradient of a separately trained classifier $p(y|x_{t})$ to steer sampling toward class $y$: the score is modified as $\nabla_{x} \log p(x_{t}) + s\,\nabla_{x} \log p(y|x_{t})$, where $s$ is the guidance scale. It requires an extra noise-robust classifier.

What is classifier-free guidance (CFG) and its main advantage?

A technique that trains a single conditional model which sometimes drops the condition (null token), then at sampling combines conditional and unconditional predictions. It avoids needing a separate classifier while still allowing controllable conditioning strength.

Write the classifier-free guidance formula for the predicted noise.

$$\tilde{\epsilon}_{\theta}(x_{t}, c) = \epsilon_{\theta}(x_{t}, \varnothing) + w\,\big(\epsilon_{\theta}(x_{t}, c) - \epsilon_{\theta}(x_{t}, \varnothing)\big),$$ where $c$ is the conditioning, $\varnothing$ the null condition, and $w$ the guidance scale.

How does the guidance scale $w$ affect classifier-free guidance outputs?

Higher $w$ increases adherence to the prompt/condition and sample fidelity but reduces diversity and can cause oversaturation or artifacts; $w = 1$ recovers ordinary conditional sampling, and $w = 0$ gives unconditional generation.

How is the unconditional model obtained in classifier-free guidance training?

By randomly replacing the conditioning $c$ with a null/empty token for a fraction of training examples (e.g., 10–20%), so the same network learns both conditional $\epsilon_{\theta}(x_t, c)$ and unconditional $\epsilon_{\theta}(x_t, \varnothing)$ predictions.

What role does a CLIP text encoder play in text-to-image models?

It converts the text prompt into semantically rich embeddings aligned with visual concepts; these embeddings condition the generator (via cross-attention) so that produced images match the described content.

How does DALL·E 2 (unCLIP) use CLIP for text-to-image generation?

A prior maps a CLIP text embedding to a CLIP image embedding, and a diffusion decoder generates the image conditioned on that image embedding. This two-stage design leverages CLIP's shared text–image latent space.

What is the core autoregressive assumption of PixelRNN and PixelCNN?

They model an image as a product of per-pixel conditionals in raster-scan order: $$p(x) = \prod_{i=1}^{n} p(x_{i} \mid x_{1}, \dots, x_{i-1}),$$ generating each pixel conditioned on all previously generated pixels.

What is the key difference between PixelRNN and PixelCNN?

PixelRNN uses recurrent layers (Row/Diagonal LSTM) to capture pixel dependencies, giving a broad receptive field but slow sequential training. PixelCNN uses masked convolutions, enabling parallel training but with a more limited (blind-spot-prone) receptive field.

Why are masked convolutions needed in PixelCNN?

To preserve the autoregressive property: the mask zeroes out access to the current and future pixels so each output depends only on already-generated pixels above and to the left, preventing information leakage from the pixel being predicted.

What is the 'blind spot' problem in PixelCNN, and how is it addressed?

Stacked masked convolutions fail to cover some context pixels, creating a blind spot in the receptive field. Gated PixelCNN fixes it by combining separate vertical and horizontal convolutional stacks so the full preceding context is captured.

How does PixelCNN model the distribution of a pixel's color value?

As a discrete categorical (softmax) distribution over the 256 possible intensity values per channel (or via a discretized mixture of logistics in PixelCNN++), trained with a cross-entropy/negative-log-likelihood loss.

Contrast training speed vs. sampling speed for autoregressive pixel models.

PixelCNN trains efficiently in parallel (all conditionals computed at once with teacher forcing), but sampling is inherently sequential — each pixel must be generated one at a time — making generation slow, on the order of $O(n)$ forward passes for $n$ pixels.

What does temperature $\tau$ do when sampling from a softmax output?

It rescales logits before softmax: $p_{i} = \dfrac{\exp(z_{i}/\tau)}{\sum_{j} \exp(z_{j}/\tau)}$. Lower $\tau$ (<1) sharpens the distribution (more deterministic/confident), while higher $\tau$ (>1) flattens it (more random/diverse).

Describe top-k sampling.

At each step, keep only the $k$ tokens with the highest probability, renormalize their probabilities to sum to 1, and sample from that truncated set. This blocks low-probability tokens while retaining some randomness.

Describe top-p (nucleus) sampling and how it differs from top-k.

Top-p keeps the smallest set of tokens whose cumulative probability exceeds a threshold $p$ (the nucleus), then samples from it. Unlike fixed-size top-k, the candidate set size adapts to the distribution's shape — small when the model is confident, larger when uncertain.

Compare greedy decoding with sampling-based strategies.

Greedy decoding always picks the highest-probability token, yielding deterministic but often repetitive, low-diversity output. Sampling (with temperature, top-k, or top-p) injects controlled randomness for more diverse and creative generations at some cost to reliability.

Rank VAEs, GANs, and diffusion models on sample quality vs. training stability.

Sample quality: diffusion $\gtrsim$ GANs $>$ VAEs (VAEs tend to be blurry). Training stability: diffusion and VAEs are stable and have well-defined likelihood-based objectives, whereas GANs are adversarial and prone to mode collapse and instability. Diffusion's main drawback is slow, multi-step sampling.