Statistics & Probability Bayesian Statistics Flashcards

24 sample cards from the Bayesian Statistics deck

Sampled from the end of the deck, so these are different cards from the ones shown on the syllabus page.

1. Define the Bayes factor $BF_{10}$ comparing model $M_1$ to model $M_0$.

   $$BF_{10} = \frac{p(y \mid M_1)}{p(y \mid M_0)} = \frac{\int p(y\mid\theta_1,M_1)p(\theta_1\mid M_1)\,d\theta_1}{\int p(y\mid\theta_0,M_0)p(\theta_0\mid M_0)\,d\theta_0}.$$ It is the ratio of marginal likelihoods (evidence).

2. How does the Bayes factor relate the prior odds to the posterior odds of two models?

   $$\underbrace{\frac{p(M_1\mid y)}{p(M_0\mid y)}}_{\text{posterior odds}} = BF_{10} \times \underbrace{\frac{p(M_1)}{p(M_0)}}_{\text{prior odds}}.$$ The Bayes factor is the multiplicative update from prior to posterior odds.

3. According to Jeffreys' scale, what does a Bayes factor $BF_{10}$ between 3 and 10 indicate?

   It indicates moderate (substantial) evidence for $M_1$ over $M_0$. Roughly: 1–3 barely worth mentioning, 3–10 moderate, 10–30 strong, 30–100 very strong, $>100$ decisive.

4. Why are Bayes factors sensitive to the choice of prior, unlike posterior estimation with lots of data?

   The marginal likelihood integrates the likelihood over the prior, so the prior's spread directly scales the evidence. Vague/diffuse priors penalize complex models (Lindley's paradox), and improper priors leave the Bayes factor undefined up to an arbitrary constant.

5. State Lindley's (Jeffreys–Lindley) paradox.

   For a point null hypothesis, a frequentist test can reject the null at a fixed significance level while the Bayes factor simultaneously favors the null, especially with large samples and diffuse priors. The two paradigms can reach opposite conclusions.

6. In Bayesian hypothesis testing, how is a point null hypothesis $H_0: \theta=\theta_0$ typically handled differently than in estimation?

   It is handled by placing a prior with a point mass (spike) on $\theta_0$ mixed with a diffuse (slab) distribution under the alternative, then computing the posterior probability of the null or the Bayes factor; a purely continuous prior would assign the point null zero probability.

7. Contrast the frequentist p-value with the Bayesian posterior probability of a hypothesis.

   A p-value is $P(\text{data as or more extreme}\mid H_0)$—the probability of the data under the null. A Bayesian posterior probability is $P(H_0\mid \text{data})$—the probability of the hypothesis given the data. They answer different questions and are not interchangeable.

8. What quantity does the Bayesian approach let you compute that frequentist NHST does not: the probability of the null being true?

   Bayesian inference gives $P(H_0 \mid y)$, the posterior probability that the null hypothesis is true, via Bayes' theorem over the model space. Frequentist NHST never assigns a probability to a hypothesis, only to data under a hypothesis.

9. What is the fundamental purpose of Markov Chain Monte Carlo (MCMC) in Bayesian inference?

   MCMC draws samples from a posterior distribution that is known only up to a normalizing constant, by constructing a Markov chain whose stationary (equilibrium) distribution is the target posterior. Sample averages then approximate posterior expectations.

10. Why is MCMC needed when the marginal likelihood $p(y)$ is intractable?

    MCMC only requires the unnormalized posterior $p(y\mid\theta)p(\theta)$ (a ratio cancels the unknown $p(y)$), so it can sample the posterior without ever computing the intractable normalizing integral.

11. Describe the Metropolis-Hastings acceptance probability for a proposal $\theta^*$ from $q(\theta^*\mid\theta)$.

    $$\alpha = \min\!\left(1,\; \frac{p(\theta^*\mid y)\,q(\theta\mid\theta^*)}{p(\theta\mid y)\,q(\theta^*\mid\theta)}\right).$$ The proposal is accepted with probability $\alpha$; the posterior ratio removes the normalizing constant.

12. For a symmetric proposal distribution, how does the Metropolis-Hastings acceptance ratio simplify?

    When $q(\theta^*\mid\theta)=q(\theta\mid\theta^*)$ (Metropolis algorithm), the proposal terms cancel: $$\alpha = \min\!\left(1,\; \frac{p(\theta^*\mid y)}{p(\theta\mid y)}\right).$$

13. What is Gibbs sampling and when is it applicable?

    Gibbs sampling updates each parameter (or block) in turn by drawing from its full conditional distribution $p(\theta_j\mid \theta_{-j}, y)$. It is applicable when these full conditionals are available in closed form and easy to sample; every proposal is accepted.

14. What defines Hamiltonian Monte Carlo (HMC) and what advantage does it offer over random-walk Metropolis?

    HMC introduces auxiliary momentum variables and simulates Hamiltonian dynamics (using gradients of the log-posterior) to propose distant states with high acceptance. It suppresses random-walk behavior, exploring high-dimensional posteriors far more efficiently.

15. What is the No-U-Turn Sampler (NUTS)?

    NUTS is an adaptive extension of HMC that automatically tunes the trajectory length by simulating until the path starts to double back (makes a U-turn), avoiding the need to hand-tune the number of leapfrog steps. It is the default sampler in Stan and PyMC.

16. Define the burn-in (warm-up) period in an MCMC run.

    Burn-in is the initial set of iterations discarded to remove dependence on the starting values, allowing the chain to reach its stationary distribution before samples are used for inference.

17. What is thinning in MCMC and why might it be used?

    Thinning keeps only every $k$-th sample to reduce autocorrelation and storage. It does not improve estimator accuracy per unit compute (it discards information) but can be useful for memory or when samples are stored/post-processed.

18. Define the Gelman-Rubin statistic $\hat{R}$ and the value indicating convergence.

    $\hat{R}$ (potential scale reduction factor) compares between-chain variance $B$ to within-chain variance $W$ across multiple chains: $$\hat{R} = \sqrt{\frac{\hat{V}}{W}}, \quad \hat{V} = \frac{N-1}{N}W + \frac{1}{N}B.$$ Values near 1 (commonly $\hat{R} < 1.01$) indicate convergence.

19. What does the effective sample size (ESS) measure in an MCMC chain?

    ESS estimates the number of independent samples equivalent to the autocorrelated MCMC draws: $$\text{ESS} = \frac{N}{1 + 2\sum_{k=1}^{\infty}\rho_k},$$ where $\rho_k$ is the lag-$k$ autocorrelation. Low ESS signals high autocorrelation and poor mixing.

20. What convergence problem does a trace plot help diagnose, and what does a well-mixed trace look like?

    A trace plot (parameter value vs. iteration) diagnoses poor mixing and non-stationarity. A well-mixed, converged chain looks like a stationary 'fuzzy caterpillar' with no trends, and overlaid multiple chains should be indistinguishable.

21. In HMC/NUTS diagnostics, what is a divergent transition and what does it indicate?

    A divergent transition occurs when the numerical leapfrog integrator's energy error becomes large, often in regions of high posterior curvature. Divergences indicate the sampler may be missing part of the posterior; they are addressed by increasing the target acceptance rate (adapt_delta) or reparameterizing.

22. What is a probabilistic programming language (PPL) and what does it automate?

    A PPL lets users specify a generative probabilistic model (priors and likelihood) in code, then automatically performs inference (e.g. MCMC, HMC/NUTS, or variational inference) to obtain the posterior, abstracting away the hand-derivation of samplers.

23. Name three widely used probabilistic programming frameworks and their key inference engines.

    Stan (NUTS/HMC via a compiled C++ backend), PyMC (NUTS/HMC and variational inference in Python), and Pyro/NumPyro (built on PyTorch/JAX, emphasizing stochastic variational inference and HMC).

24. What is variational inference (VI) and how does it differ from MCMC as offered in probabilistic programming tools?

    VI approximates the posterior by choosing the member of a tractable family $q(\theta)$ that minimizes the KL divergence to the true posterior (maximizing the ELBO). Unlike MCMC, it is an optimization method: faster and scalable but yielding an approximate (often variance-underestimating) posterior rather than asymptotically exact samples.

What this deck covers

Bayesian Statistics flashcards FAQ