Statistics & Probability Bayesian Statistics Syllabus

Bayesian Statistics syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Bayesian Statistics in Statistics & Probability, not a summary of it.

1. 1

   Bayesian Foundations

   4 topics

   • Frequentist vs Bayesian Paradigms
   • Priors, Likelihoods and Posteriors
   • Conjugate Priors
   • Posterior Predictive Distribution
2. 2

   Bayesian Inference

   3 topics

   • Credible Intervals
   • Bayes Factors
   • Bayesian vs Classical Hypothesis Testing
3. 3

   Computational Bayesian Methods

   3 topics

   • Markov Chain Monte Carlo (MCMC)
   • Convergence Diagnostics
   • Probabilistic Programming

Bayesian Statistics flashcards for Statistics & Probability

21 of 50 cards from the Bayesian Statistics deck — real questions with worked answers.

1. What is the core philosophical difference between the frequentist and Bayesian interpretations of probability?

   Frequentists interpret probability as the long-run relative frequency of an event over repeated trials, treating parameters as fixed unknown constants. Bayesians interpret probability as a degree of belief, treating parameters as random variables with probability distributions.

2. In the frequentist paradigm, is a parameter $\theta$ treated as random or fixed, and what is treated as random instead?

   In the frequentist paradigm $\theta$ is a fixed (unknown) constant; the data are treated as random. In the Bayesian paradigm $\theta$ is random (has a distribution) and the observed data are fixed.

3. State Bayes' theorem for a parameter $\theta$ given data $y$ in its standard proportional form.

   $$p(\theta \mid y) = \frac{p(y \mid \theta)\, p(\theta)}{p(y)} \propto p(y \mid \theta)\, p(\theta)$$ i.e. posterior $\propto$ likelihood $\times$ prior.

4. In Bayes' theorem, name the four components: $p(\theta)$, $p(y\mid\theta)$, $p(\theta\mid y)$, and $p(y)$.

   $p(\theta)$ is the prior, $p(y\mid\theta)$ is the likelihood, $p(\theta\mid y)$ is the posterior, and $p(y)$ is the marginal likelihood (evidence).

5. What is the marginal likelihood (evidence) $p(y)$ and how is it computed for a continuous parameter?

   It is the normalizing constant of the posterior, obtained by integrating the joint over $\theta$: $$p(y) = \int p(y \mid \theta)\, p(\theta)\, d\theta.$$ It makes the posterior integrate to 1.

6. Distinguish the likelihood $p(y\mid\theta)$ viewed as a function of the data versus as a function of $\theta$.

   As a function of $y$ (with $\theta$ fixed) it is a probability distribution that integrates to 1. As a function of $\theta$ (with $y$ fixed) it is the likelihood $L(\theta)$, which need not integrate to 1 and is not a probability distribution over $\theta$.

7. What is an informative prior versus a non-informative (vague) prior?

   An informative prior encodes substantial prior knowledge, concentrating probability in a region of parameter space. A non-informative (vague/diffuse) prior expresses minimal prior knowledge, e.g. a very wide distribution, aiming to let the data dominate the posterior.

8. Define an improper prior and give an example.

   An improper prior does not integrate to a finite value (it is not a valid probability distribution), e.g. a uniform prior $p(\theta) \propto 1$ over $(-\infty,\infty)$. It can still yield a proper posterior if $p(y)$ is finite.

9. What is a Jeffreys prior and what key property does it have?

   The Jeffreys prior is $p(\theta) \propto \sqrt{\det I(\theta)}$, where $I(\theta)$ is the Fisher information. Its key property is invariance under reparameterization of $\theta$.

10. For a single parameter, write the Jeffreys prior in terms of the Fisher information $I(\theta)$.

    $$p(\theta) \propto \sqrt{I(\theta)}, \qquad I(\theta) = -\,\mathbb{E}\!\left[\frac{\partial^{2}}{\partial\theta^{2}} \log p(y\mid\theta)\right].$$

11. Define a conjugate prior.

    A prior is conjugate to a likelihood if the resulting posterior belongs to the same distributional family as the prior. This yields closed-form posterior updates.

12. What is the conjugate prior for the mean of a normal likelihood with known variance, and what family is the posterior?

    The conjugate prior is a normal (Gaussian) distribution; the posterior for the mean is also normal. The Normal family is self-conjugate for the mean with known variance.

13. For a Binomial likelihood with success probability $\theta$, what is the conjugate prior and the posterior after observing $s$ successes in $n$ trials?

    The conjugate prior is $\text{Beta}(\alpha,\beta)$. The posterior is $$\theta \mid y \sim \text{Beta}(\alpha + s,\; \beta + n - s).$$

14. For a Poisson likelihood with rate $\lambda$, what is the conjugate prior and the posterior after observing counts $y_1,\dots,y_n$?

    The conjugate prior is $\text{Gamma}(\alpha,\beta)$ (shape–rate). The posterior is $$\lambda \mid y \sim \text{Gamma}\!\left(\alpha + \sum_{i=1}^{n} y_i,\; \beta + n\right).$$

15. What is the conjugate prior for the probabilities of a Multinomial (categorical) likelihood?

    The Dirichlet distribution. If the prior is $\text{Dirichlet}(\alpha_1,\dots,\alpha_k)$ and counts are $n_1,\dots,n_k$, the posterior is $\text{Dirichlet}(\alpha_1+n_1,\dots,\alpha_k+n_k)$.

16. For an exponential (or Poisson rate) parameter, why is the Gamma distribution a convenient conjugate prior?

    The Gamma is conjugate to the exponential and Poisson likelihoods, so the posterior remains Gamma; its shape and rate parameters update by simple addition of counts/observations, giving closed-form inference.

17. Interpret the Beta$(\alpha,\beta)$ prior parameters as pseudo-counts.

    $\alpha$ acts as a prior count of successes and $\beta$ as a prior count of failures. The prior effectively contributes $\alpha+\beta$ pseudo-observations, with prior mean $\frac{\alpha}{\alpha+\beta}$.

18. Show that the posterior mean of a Beta-Binomial model is a weighted average of the prior mean and the sample proportion.

    With prior $\text{Beta}(\alpha,\beta)$ and data $s$ successes in $n$ trials, $$\mathbb{E}[\theta\mid y] = \frac{\alpha+s}{\alpha+\beta+n},$$ a weighted average of the prior mean $\frac{\alpha}{\alpha+\beta}$ and the MLE $\frac{s}{n}$, with weights depending on $\alpha+\beta$ and $n$.

19. Define the posterior predictive distribution for a new observation $\tilde{y}$.

    $$p(\tilde{y} \mid y) = \int p(\tilde{y} \mid \theta)\, p(\theta \mid y)\, d\theta.$$ It predicts new data by averaging the likelihood over the posterior of $\theta$.

20. How does the posterior predictive distribution differ from simply plugging in a point estimate $\hat{\theta}$?

    The posterior predictive integrates over the full posterior uncertainty in $\theta$, so it accounts for parameter uncertainty and is typically wider (more dispersed) than the plug-in predictive $p(\tilde{y}\mid\hat{\theta})$.

21. What is the prior predictive distribution and how does it relate to the marginal likelihood?

    The prior predictive $p(\tilde{y}) = \int p(\tilde{y}\mid\theta)\,p(\theta)\,d\theta$ is the distribution of data before observing any; evaluated at the observed data it equals the marginal likelihood (evidence) $p(y)$.

See more Bayesian Statistics flashcards →

Planning Bayesian Statistics for Statistics & Probability

Bayesian Statistics (Statistics & Probability) FAQ