Statistics & Probability Probability Theory Syllabus

Define independent events.

Two events $A$ and $B$ are independent if the occurrence of one does not affect the probability of the other: $P(A \cap B) = P(A)\,P(B)$, equivalently $P(A \mid B) = P(A)$.

Can two events with nonzero probability be both mutually exclusive and independent? Explain.

No. If $A$ and $B$ are mutually exclusive, $P(A \cap B) = 0$, but independence requires $P(A \cap B) = P(A)P(B) > 0$ when both are nonzero. So the two properties are incompatible for events of positive probability.

Contrast mutually exclusive events with independent events.

Mutually exclusive means the events cannot co-occur ($P(A\cap B)=0$); knowing one occurred tells you the other did not (highly dependent). Independent means one event's occurrence gives no information about the other ($P(A\cap B)=P(A)P(B)$).

Define conditional probability and give its formula.

The conditional probability of $A$ given $B$ (with $P(B) > 0$) is $$P(A \mid B) = \frac{P(A \cap B)}{P(B)}.$$ It is the probability of $A$ restricted to the outcomes where $B$ occurs.

What is the standard notation $P(A \mid B)$ read as, and what does it mean?

It is read 'the probability of $A$ given $B$' and denotes the probability that event $A$ occurs given that event $B$ has already occurred.

Give the three equivalent conditions that characterize independence of events $A$ and $B$.

$A$ and $B$ are independent iff any one of these holds: $P(A \cap B) = P(A)P(B)$, or $P(A \mid B) = P(A)$, or $P(B \mid A) = P(B)$ (assuming the conditioning events have positive probability).

What does it mean for three events $A$, $B$, $C$ to be mutually independent?

They must satisfy pairwise independence AND $P(A \cap B \cap C) = P(A)P(B)P(C)$. Pairwise independence alone is not sufficient for mutual independence.

State the Law of Total Probability for a partition $B_1, \dots, B_n$ of the sample space.

If $B_1,\dots,B_n$ are mutually exclusive and exhaustive with $P(B_i)>0$, then for any event $A$: $$P(A) = \sum_{i=1}^{n} P(A \mid B_i)\,P(B_i).$$

State the two-event form of the Law of Total Probability.

For an event $B$ and its complement $B^{c}$: $$P(A) = P(A \mid B)P(B) + P(A \mid B^{c})P(B^{c}).$$

What must a set of events satisfy to form a valid partition of the sample space?

They must be mutually exclusive (no overlap: $B_i \cap B_j = \varnothing$ for $i \neq j$) and collectively exhaustive (their union is $S$, so $\bigcup_i B_i = S$).

What is a tree diagram used for in probability?

A tree diagram represents a sequence of events with branches; each branch is labeled with a (conditional) probability. Multiplying probabilities along a path gives the probability of that combined outcome, and paths for the same event are added.

In a probability tree, how do you find the probability of a specific final outcome?

Multiply the probabilities along the branches from the root to that outcome. Each branch after the first represents a conditional probability given the previous branches.

In a probability tree, what must the probabilities on branches emanating from a single node sum to?

They must sum to 1, since the branches from a node represent all mutually exclusive possibilities at that stage.

Define prior probability.

A prior probability, $P(H)$, is the probability assigned to a hypothesis or event before observing new evidence — it reflects initial belief based on existing information.

Define posterior probability.

A posterior probability, $P(H \mid E)$, is the updated probability of a hypothesis $H$ after taking into account new evidence $E$. It is computed via Bayes' theorem.

State Bayes' Theorem for events $A$ and $B$.

$$P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}, \quad P(B) > 0.$$

Derive Bayes' Theorem from the definition of conditional probability.

By definition $P(A\mid B) = \frac{P(A\cap B)}{P(B)}$ and $P(B\mid A) = \frac{P(A\cap B)}{P(A)}$. From the second, $P(A\cap B) = P(B\mid A)P(A)$. Substituting into the first gives $$P(A\mid B) = \frac{P(B\mid A)P(A)}{P(B)}.$$

State the expanded form of Bayes' Theorem using the Law of Total Probability in the denominator.

For a partition $A_1,\dots,A_n$: $$P(A_k \mid B) = \frac{P(B \mid A_k)P(A_k)}{\sum_{i=1}^{n} P(B \mid A_i)P(A_i)}.$$

In Bayes' Theorem, identify the prior, likelihood, and posterior in $P(H\mid E) = \frac{P(E\mid H)P(H)}{P(E)}$.

$P(H)$ is the prior, $P(E\mid H)$ is the likelihood (probability of the evidence given the hypothesis), $P(E)$ is the marginal/evidence, and $P(H\mid E)$ is the posterior.

Give a classic application of Bayes' Theorem in medical testing.

Given a disease with prevalence $P(D)$ and a test with sensitivity $P(+\mid D)$ and false-positive rate $P(+\mid D^{c})$, Bayes' theorem finds the probability a person actually has the disease given a positive test: $$P(D\mid +) = \frac{P(+\mid D)P(D)}{P(+\mid D)P(D) + P(+\mid D^{c})P(D^{c})}.$$

What is the base rate fallacy?

The base rate fallacy is the error of ignoring the prior probability (base rate) of an event and over-relying on specific test/evidence information, leading to overestimation of the posterior probability — especially for rare events.

Why can a highly accurate test still yield a low posterior probability for a rare disease (base rate fallacy)?

When the disease is rare, the large healthy population produces many false positives that can outnumber true positives. Even with high sensitivity, $P(D\mid +)$ stays low because the small prior $P(D)$ dominates the calculation.

Define a random variable.

A random variable is a function that assigns a real number to each outcome in the sample space of a random experiment, typically denoted by a capital letter such as $X$.

Distinguish a discrete random variable from a continuous random variable.

A discrete random variable takes a finite or countably infinite set of distinct values (e.g. counts). A continuous random variable can take any value in an interval of real numbers (e.g. measurements like height or time).

Define the probability mass function (PMF) of a discrete random variable.

The PMF is $p(x) = P(X = x)$, giving the probability that the discrete random variable $X$ equals each value $x$. It must satisfy $p(x) \geq 0$ and $\sum_{x} p(x) = 1$.

Define the probability density function (PDF) of a continuous random variable.

A PDF $f(x)$ satisfies $f(x) \geq 0$ and $\int_{-\infty}^{\infty} f(x)\,dx = 1$. Probabilities are areas: $$P(a \leq X \leq b) = \int_{a}^{b} f(x)\,dx.$$

For a continuous random variable, what is $P(X = c)$ for any single value $c$?

$P(X = c) = 0$. For a continuous variable the probability of any exact single value is zero because it corresponds to zero area under the density curve; only intervals carry positive probability.

Compare how probabilities are obtained from a PMF versus a PDF.

For a discrete variable, probability comes directly from the PMF value $P(X=x)=p(x)$ and is summed over values. For a continuous variable, the PDF value $f(x)$ is not a probability; probability is found by integrating $f(x)$ over an interval.

What is the normalization (total probability) condition for a PMF and for a PDF?

PMF: $\sum_{x} p(x) = 1$. PDF: $\int_{-\infty}^{\infty} f(x)\,dx = 1$. In both cases the total probability over all outcomes equals 1.

Define the cumulative distribution function (CDF) of a random variable $X$.

The CDF is $F(x) = P(X \leq x)$. For a discrete variable $F(x) = \sum_{t \leq x} p(t)$; for a continuous variable $F(x) = \int_{-\infty}^{x} f(t)\,dt$, and $f(x) = F'(x)$ where the derivative exists.