Examius
🌍 Statistics & Probability · subject
Statistics & Probability Bayesian Statistics Syllabus
The Bayesian Statistics syllabus for Statistics & Probability in full: 3 chapters and 10 topics, every one listed below, plus 50 Bayesian Statistics flashcards with answers.
Bayesian Statistics syllabus: all 3 chapters
Expand any chapter to see its topics and sub-topics.
-
Bayesian Foundations
4 topics- Frequentist vs Bayesian Paradigms
- Priors, Likelihoods and Posteriors
- Conjugate Priors
- Posterior Predictive Distribution
-
Bayesian Inference
3 topics- Credible Intervals
- Bayes Factors
- Bayesian vs Classical Hypothesis Testing
-
Computational Bayesian Methods
3 topics- Markov Chain Monte Carlo (MCMC)
- Convergence Diagnostics
- Probabilistic Programming
Statistics & Probability Bayesian Statistics flashcards — 50 questions with answers
What is the core philosophical difference between the frequentist and Bayesian interpretations of probability?
Frequentists interpret probability as the long-run relative frequency of an event over repeated trials, treating parameters as fixed unknown constants. Bayesians interpret probability as a degree of belief, treating parameters as random variables with probability distributions.
In the frequentist paradigm, is a parameter $\theta$ treated as random or fixed, and what is treated as random instead?
In the frequentist paradigm $\theta$ is a fixed (unknown) constant; the data are treated as random. In the Bayesian paradigm $\theta$ is random (has a distribution) and the observed data are fixed.
State Bayes' theorem for a parameter $\theta$ given data $y$ in its standard proportional form.
$$p(\theta \mid y) = \frac{p(y \mid \theta)\, p(\theta)}{p(y)} \propto p(y \mid \theta)\, p(\theta)$$ i.e. posterior $\propto$ likelihood $\times$ prior.
In Bayes' theorem, name the four components: $p(\theta)$, $p(y\mid\theta)$, $p(\theta\mid y)$, and $p(y)$.
$p(\theta)$ is the prior, $p(y\mid\theta)$ is the likelihood, $p(\theta\mid y)$ is the posterior, and $p(y)$ is the marginal likelihood (evidence).
What is the marginal likelihood (evidence) $p(y)$ and how is it computed for a continuous parameter?
It is the normalizing constant of the posterior, obtained by integrating the joint over $\theta$: $$p(y) = \int p(y \mid \theta)\, p(\theta)\, d\theta.$$ It makes the posterior integrate to 1.
Distinguish the likelihood $p(y\mid\theta)$ viewed as a function of the data versus as a function of $\theta$.
As a function of $y$ (with $\theta$ fixed) it is a probability distribution that integrates to 1. As a function of $\theta$ (with $y$ fixed) it is the likelihood $L(\theta)$, which need not integrate to 1 and is not a probability distribution over $\theta$.
What is an informative prior versus a non-informative (vague) prior?
An informative prior encodes substantial prior knowledge, concentrating probability in a region of parameter space. A non-informative (vague/diffuse) prior expresses minimal prior knowledge, e.g. a very wide distribution, aiming to let the data dominate the posterior.
Define an improper prior and give an example.
An improper prior does not integrate to a finite value (it is not a valid probability distribution), e.g. a uniform prior $p(\theta) \propto 1$ over $(-\infty,\infty)$. It can still yield a proper posterior if $p(y)$ is finite.
What is a Jeffreys prior and what key property does it have?
The Jeffreys prior is $p(\theta) \propto \sqrt{\det I(\theta)}$, where $I(\theta)$ is the Fisher information. Its key property is invariance under reparameterization of $\theta$.
For a single parameter, write the Jeffreys prior in terms of the Fisher information $I(\theta)$.
$$p(\theta) \propto \sqrt{I(\theta)}, \qquad I(\theta) = -\,\mathbb{E}\!\left[\frac{\partial^{2}}{\partial\theta^{2}} \log p(y\mid\theta)\right].$$
Define a conjugate prior.
A prior is conjugate to a likelihood if the resulting posterior belongs to the same distributional family as the prior. This yields closed-form posterior updates.
What is the conjugate prior for the mean of a normal likelihood with known variance, and what family is the posterior?
The conjugate prior is a normal (Gaussian) distribution; the posterior for the mean is also normal. The Normal family is self-conjugate for the mean with known variance.
For a Binomial likelihood with success probability $\theta$, what is the conjugate prior and the posterior after observing $s$ successes in $n$ trials?
The conjugate prior is $\text{Beta}(\alpha,\beta)$. The posterior is $$\theta \mid y \sim \text{Beta}(\alpha + s,\; \beta + n - s).$$
For a Poisson likelihood with rate $\lambda$, what is the conjugate prior and the posterior after observing counts $y_1,\dots,y_n$?
The conjugate prior is $\text{Gamma}(\alpha,\beta)$ (shape–rate). The posterior is $$\lambda \mid y \sim \text{Gamma}\!\left(\alpha + \sum_{i=1}^{n} y_i,\; \beta + n\right).$$
What is the conjugate prior for the probabilities of a Multinomial (categorical) likelihood?
The Dirichlet distribution. If the prior is $\text{Dirichlet}(\alpha_1,\dots,\alpha_k)$ and counts are $n_1,\dots,n_k$, the posterior is $\text{Dirichlet}(\alpha_1+n_1,\dots,\alpha_k+n_k)$.
For an exponential (or Poisson rate) parameter, why is the Gamma distribution a convenient conjugate prior?
The Gamma is conjugate to the exponential and Poisson likelihoods, so the posterior remains Gamma; its shape and rate parameters update by simple addition of counts/observations, giving closed-form inference.
Interpret the Beta$(\alpha,\beta)$ prior parameters as pseudo-counts.
$\alpha$ acts as a prior count of successes and $\beta$ as a prior count of failures. The prior effectively contributes $\alpha+\beta$ pseudo-observations, with prior mean $\frac{\alpha}{\alpha+\beta}$.
Show that the posterior mean of a Beta-Binomial model is a weighted average of the prior mean and the sample proportion.
With prior $\text{Beta}(\alpha,\beta)$ and data $s$ successes in $n$ trials, $$\mathbb{E}[\theta\mid y] = \frac{\alpha+s}{\alpha+\beta+n},$$ a weighted average of the prior mean $\frac{\alpha}{\alpha+\beta}$ and the MLE $\frac{s}{n}$, with weights depending on $\alpha+\beta$ and $n$.
Define the posterior predictive distribution for a new observation $\tilde{y}$.
$$p(\tilde{y} \mid y) = \int p(\tilde{y} \mid \theta)\, p(\theta \mid y)\, d\theta.$$ It predicts new data by averaging the likelihood over the posterior of $\theta$.
How does the posterior predictive distribution differ from simply plugging in a point estimate $\hat{\theta}$?
The posterior predictive integrates over the full posterior uncertainty in $\theta$, so it accounts for parameter uncertainty and is typically wider (more dispersed) than the plug-in predictive $p(\tilde{y}\mid\hat{\theta})$.
Show the other 30 Bayesian Statistics flashcards
What is the prior predictive distribution and how does it relate to the marginal likelihood?
The prior predictive $p(\tilde{y}) = \int p(\tilde{y}\mid\theta)\,p(\theta)\,d\theta$ is the distribution of data before observing any; evaluated at the observed data it equals the marginal likelihood (evidence) $p(y)$.
For the Beta-Binomial model, what is the posterior predictive distribution of the number of successes in $m$ future trials?
It is the Beta-Binomial distribution: with posterior $\text{Beta}(\alpha',\beta')$, $$p(\tilde{s}\mid y) = \binom{m}{\tilde{s}} \frac{B(\tilde{s}+\alpha',\, m-\tilde{s}+\beta')}{B(\alpha',\beta')}.$$
Define a $100(1-\alpha)\%$ credible interval.
An interval $[L,U]$ such that the posterior probability that $\theta$ lies within it is $1-\alpha$: $$P(L \leq \theta \leq U \mid y) = 1-\alpha.$$
Contrast the interpretation of a Bayesian credible interval with a frequentist confidence interval.
A credible interval directly states there is a $1-\alpha$ probability the parameter lies in it, given the data. A confidence interval states that under repeated sampling $100(1-\alpha)\%$ of such intervals would contain the fixed true parameter; the probability statement is about the procedure, not this specific interval.
What is a Highest Posterior Density (HPD) interval and how does it differ from an equal-tailed interval?
An HPD interval is the shortest interval containing $1-\alpha$ posterior probability such that every point inside has higher density than any point outside. An equal-tailed interval simply cuts $\alpha/2$ probability from each tail; the two coincide only for symmetric posteriors.
For a skewed (asymmetric) posterior, which type of credible interval is generally shorter: HPD or equal-tailed?
The HPD interval is generally shorter (or equal) because it is defined as the shortest interval of given posterior mass, whereas the equal-tailed interval is not optimized for length.
Define the Bayes factor $BF_{10}$ comparing model $M_1$ to model $M_0$.
$$BF_{10} = \frac{p(y \mid M_1)}{p(y \mid M_0)} = \frac{\int p(y\mid\theta_1,M_1)p(\theta_1\mid M_1)\,d\theta_1}{\int p(y\mid\theta_0,M_0)p(\theta_0\mid M_0)\,d\theta_0}.$$ It is the ratio of marginal likelihoods (evidence).
How does the Bayes factor relate the prior odds to the posterior odds of two models?
$$\underbrace{\frac{p(M_1\mid y)}{p(M_0\mid y)}}_{\text{posterior odds}} = BF_{10} \times \underbrace{\frac{p(M_1)}{p(M_0)}}_{\text{prior odds}}.$$ The Bayes factor is the multiplicative update from prior to posterior odds.
According to Jeffreys' scale, what does a Bayes factor $BF_{10}$ between 3 and 10 indicate?
It indicates moderate (substantial) evidence for $M_1$ over $M_0$. Roughly: 1–3 barely worth mentioning, 3–10 moderate, 10–30 strong, 30–100 very strong, $>100$ decisive.
Why are Bayes factors sensitive to the choice of prior, unlike posterior estimation with lots of data?
The marginal likelihood integrates the likelihood over the prior, so the prior's spread directly scales the evidence. Vague/diffuse priors penalize complex models (Lindley's paradox), and improper priors leave the Bayes factor undefined up to an arbitrary constant.
State Lindley's (Jeffreys–Lindley) paradox.
For a point null hypothesis, a frequentist test can reject the null at a fixed significance level while the Bayes factor simultaneously favors the null, especially with large samples and diffuse priors. The two paradigms can reach opposite conclusions.
In Bayesian hypothesis testing, how is a point null hypothesis $H_0: \theta=\theta_0$ typically handled differently than in estimation?
It is handled by placing a prior with a point mass (spike) on $\theta_0$ mixed with a diffuse (slab) distribution under the alternative, then computing the posterior probability of the null or the Bayes factor; a purely continuous prior would assign the point null zero probability.
Contrast the frequentist p-value with the Bayesian posterior probability of a hypothesis.
A p-value is $P(\text{data as or more extreme}\mid H_0)$—the probability of the data under the null. A Bayesian posterior probability is $P(H_0\mid \text{data})$—the probability of the hypothesis given the data. They answer different questions and are not interchangeable.
What quantity does the Bayesian approach let you compute that frequentist NHST does not: the probability of the null being true?
Bayesian inference gives $P(H_0 \mid y)$, the posterior probability that the null hypothesis is true, via Bayes' theorem over the model space. Frequentist NHST never assigns a probability to a hypothesis, only to data under a hypothesis.
What is the fundamental purpose of Markov Chain Monte Carlo (MCMC) in Bayesian inference?
MCMC draws samples from a posterior distribution that is known only up to a normalizing constant, by constructing a Markov chain whose stationary (equilibrium) distribution is the target posterior. Sample averages then approximate posterior expectations.
Why is MCMC needed when the marginal likelihood $p(y)$ is intractable?
MCMC only requires the unnormalized posterior $p(y\mid\theta)p(\theta)$ (a ratio cancels the unknown $p(y)$), so it can sample the posterior without ever computing the intractable normalizing integral.
Describe the Metropolis-Hastings acceptance probability for a proposal $\theta^*$ from $q(\theta^*\mid\theta)$.
$$\alpha = \min\!\left(1,\; \frac{p(\theta^*\mid y)\,q(\theta\mid\theta^*)}{p(\theta\mid y)\,q(\theta^*\mid\theta)}\right).$$ The proposal is accepted with probability $\alpha$; the posterior ratio removes the normalizing constant.
For a symmetric proposal distribution, how does the Metropolis-Hastings acceptance ratio simplify?
When $q(\theta^*\mid\theta)=q(\theta\mid\theta^*)$ (Metropolis algorithm), the proposal terms cancel: $$\alpha = \min\!\left(1,\; \frac{p(\theta^*\mid y)}{p(\theta\mid y)}\right).$$
What is Gibbs sampling and when is it applicable?
Gibbs sampling updates each parameter (or block) in turn by drawing from its full conditional distribution $p(\theta_j\mid \theta_{-j}, y)$. It is applicable when these full conditionals are available in closed form and easy to sample; every proposal is accepted.
What defines Hamiltonian Monte Carlo (HMC) and what advantage does it offer over random-walk Metropolis?
HMC introduces auxiliary momentum variables and simulates Hamiltonian dynamics (using gradients of the log-posterior) to propose distant states with high acceptance. It suppresses random-walk behavior, exploring high-dimensional posteriors far more efficiently.
What is the No-U-Turn Sampler (NUTS)?
NUTS is an adaptive extension of HMC that automatically tunes the trajectory length by simulating until the path starts to double back (makes a U-turn), avoiding the need to hand-tune the number of leapfrog steps. It is the default sampler in Stan and PyMC.
Define the burn-in (warm-up) period in an MCMC run.
Burn-in is the initial set of iterations discarded to remove dependence on the starting values, allowing the chain to reach its stationary distribution before samples are used for inference.
What is thinning in MCMC and why might it be used?
Thinning keeps only every $k$-th sample to reduce autocorrelation and storage. It does not improve estimator accuracy per unit compute (it discards information) but can be useful for memory or when samples are stored/post-processed.
Define the Gelman-Rubin statistic $\hat{R}$ and the value indicating convergence.
$\hat{R}$ (potential scale reduction factor) compares between-chain variance $B$ to within-chain variance $W$ across multiple chains: $$\hat{R} = \sqrt{\frac{\hat{V}}{W}}, \quad \hat{V} = \frac{N-1}{N}W + \frac{1}{N}B.$$ Values near 1 (commonly $\hat{R} < 1.01$) indicate convergence.
What does the effective sample size (ESS) measure in an MCMC chain?
ESS estimates the number of independent samples equivalent to the autocorrelated MCMC draws: $$\text{ESS} = \frac{N}{1 + 2\sum_{k=1}^{\infty}\rho_k},$$ where $\rho_k$ is the lag-$k$ autocorrelation. Low ESS signals high autocorrelation and poor mixing.
What convergence problem does a trace plot help diagnose, and what does a well-mixed trace look like?
A trace plot (parameter value vs. iteration) diagnoses poor mixing and non-stationarity. A well-mixed, converged chain looks like a stationary 'fuzzy caterpillar' with no trends, and overlaid multiple chains should be indistinguishable.
In HMC/NUTS diagnostics, what is a divergent transition and what does it indicate?
A divergent transition occurs when the numerical leapfrog integrator's energy error becomes large, often in regions of high posterior curvature. Divergences indicate the sampler may be missing part of the posterior; they are addressed by increasing the target acceptance rate (adapt_delta) or reparameterizing.
What is a probabilistic programming language (PPL) and what does it automate?
A PPL lets users specify a generative probabilistic model (priors and likelihood) in code, then automatically performs inference (e.g. MCMC, HMC/NUTS, or variational inference) to obtain the posterior, abstracting away the hand-derivation of samplers.
Name three widely used probabilistic programming frameworks and their key inference engines.
Stan (NUTS/HMC via a compiled C++ backend), PyMC (NUTS/HMC and variational inference in Python), and Pyro/NumPyro (built on PyTorch/JAX, emphasizing stochastic variational inference and HMC).
What is variational inference (VI) and how does it differ from MCMC as offered in probabilistic programming tools?
VI approximates the posterior by choosing the member of a tractable family $q(\theta)$ that minimizes the KL divergence to the true posterior (maximizing the ELBO). Unlike MCMC, it is an optimization method: faster and scalable but yielding an approximate (often variance-underestimating) posterior rather than asymptotically exact samples.
Statistics & Probability Bayesian Statistics FAQ
What is in the Statistics & Probability Bayesian Statistics syllabus?
Statistics & Probability Bayesian Statistics is split into 3 chapters: Bayesian Foundations, Bayesian Inference and Computational Bayesian Methods. Together they contain 10 topics.
How many chapters are in Statistics & Probability Bayesian Statistics?
3 chapters, containing 10 topics. The largest chapter is Bayesian Foundations, with 4 topics.
Are there flashcards for Statistics & Probability Bayesian Statistics?
Yes — a 50-card Bayesian Statistics deck, printed in full in the flashcards section of this page and available in the Examius app with spaced repetition. The free plan covers one exam of your choice, including its flashcards; Examius Pro, a one-time purchase, unlocks every other exam and deck.