Statistics & Probability Statistical Reasoning & Communication Syllabus

How does the False Discovery Rate (FDR) approach differ from Bonferroni?

Bonferroni controls the probability of any false positive (family-wise error rate), which is very conservative. FDR (e.g., Benjamini-Hochberg) controls the expected proportion of false positives among rejected hypotheses, giving more power with many tests.

What is regression to the mean?

The tendency for extreme observations on one measurement to be followed by values closer to the average on a second measurement, due to the random component of the extreme value. It occurs whenever two measures are imperfectly correlated.

How can regression to the mean create a false impression that an intervention worked?

If subjects are selected because they scored extremely (e.g., very sick, very low), their follow-up scores naturally move toward the mean even without any treatment. The improvement may be misattributed to the intervention rather than statistical regression.

What experimental design guards against mistaking regression to the mean for a treatment effect?

A randomized controlled trial with a comparison/control group. Both groups regress toward the mean equally, so any difference between treated and control groups isolates the true treatment effect.

What is the ecological fallacy?

Drawing conclusions about individuals based on aggregate (group-level) statistics. A correlation observed at the group level need not hold—and may even reverse—at the individual level.

Give an example of the ecological fallacy.

Finding that regions with more immigrants have higher literacy does not mean immigrants themselves are more literate; the aggregate correlation may reflect that immigrants settled in already-literate regions. Group-level association misapplied to individuals.

What is the atomistic fallacy, the converse of the ecological fallacy?

The atomistic (or individualistic) fallacy is inferring group- or population-level relationships from individual-level data, ignoring context and aggregation effects that operate at the higher level.

Why does correlation not imply causation?

An observed association between $X$ and $Y$ can arise from $X$ causing $Y$, $Y$ causing $X$ (reverse causation), a confounder $Z$ causing both, selection bias, or coincidence. Correlation alone cannot distinguish these mechanisms.

Name the three requirements traditionally needed to infer causation.

(1) Association/covariation between cause and effect; (2) temporal precedence—the cause precedes the effect; (3) elimination of plausible alternative explanations (no confounding).

What is reverse causation, and how does it confound correlation?

Reverse causation occurs when the presumed effect actually causes the presumed cause (e.g., 'stress causes illness' when illness may cause stress). The direction of the causal arrow is opposite to what is assumed, yet the correlation looks identical.

What is a spurious correlation?

A correlation between two variables that is not due to any direct causal link, typically produced by a common cause (confounder) or coincidence. Example: ice cream sales and drownings correlate because both rise with hot weather.

In the potential outcomes (Rubin causal) framework, what are $Y_i(1)$ and $Y_i(0)$?

For unit $i$, $Y_i(1)$ is the potential outcome if treated and $Y_i(0)$ is the potential outcome if not treated. The individual causal effect is $Y_i(1) - Y_i(0)$.

What is the 'fundamental problem of causal inference'?

For any single unit we can observe only one of its potential outcomes—either $Y_i(1)$ or $Y_i(0)$, never both—so the individual causal effect $Y_i(1)-Y_i(0)$ can never be directly observed. Causal inference relies on comparing groups instead.

Define the Average Treatment Effect (ATE) in potential outcomes notation.

$$ATE = E[Y_i(1) - Y_i(0)] = E[Y_i(1)] - E[Y_i(0)]$$ the average difference in potential outcomes across the population.

What is the SUTVA assumption in the potential outcomes framework?

Stable Unit Treatment Value Assumption: (1) no interference—one unit's treatment does not affect another unit's outcome; (2) no hidden variations of treatment—the treatment is well-defined and identical across units.

What is a confounding variable?

A variable that is associated with both the treatment/exposure and the outcome and is not on the causal pathway between them. It creates a spurious (or distorted) association between exposure and outcome if not controlled.

Name three methods for controlling confounding in an analysis.

(1) Randomization (in experiments); (2) restriction or matching on the confounder; (3) statistical adjustment such as stratification or multivariable regression.

Why should you NOT control for a variable on the causal pathway (a mediator)?

Adjusting for a mediator blocks part of the true causal effect you are trying to estimate, biasing the total effect toward zero (over-adjustment bias). Only the direct effect would remain.

What is a collider, and why can controlling for it induce bias?

A collider is a variable caused by two others ($A \to C \leftarrow B$). Conditioning on (controlling for) a collider opens a spurious path between $A$ and $B$, creating an association where none existed—known as collider/selection bias.

Why is randomization the defining feature of a randomized controlled trial (RCT)?

Random assignment makes treatment independent of all baseline characteristics (measured and unmeasured), balancing confounders across groups in expectation. This licenses a causal interpretation of the group difference.

What roles do blinding and a control/placebo group play in an RCT?

A control (often placebo) group provides the counterfactual comparison. Blinding participants and assessors prevents expectation and measurement biases (e.g., the placebo effect), so observed differences reflect the treatment itself.

What is 'intention-to-treat' analysis and why is it used?

Intention-to-treat analyzes participants in the groups they were randomized to, regardless of whether they adhered to or completed the treatment. It preserves the benefits of randomization and avoids bias from non-random dropout or crossover.

Name three observational methods used to estimate causal effects when randomization is impossible.

(1) Instrumental variables; (2) regression discontinuity design; (3) difference-in-differences. (Others: propensity score matching, natural experiments.)

How does a difference-in-differences (DiD) design estimate a causal effect?

It compares the change in outcomes over time for a treated group to the change for a control group: $$DiD = (Y_{treat,after}-Y_{treat,before}) - (Y_{ctrl,after}-Y_{ctrl,before})$$ relying on the parallel trends assumption.

What is the key assumption behind a regression discontinuity design?

Units just above and just below a cutoff/threshold on a running variable are comparable (as-if randomized), so any jump in the outcome at the cutoff is attributable to the treatment assigned by that cutoff.

What two conditions must a valid instrumental variable satisfy?

(1) Relevance—the instrument is correlated with the treatment; (2) Exclusion/exogeneity—the instrument affects the outcome only through the treatment and is uncorrelated with confounders.

What is pre-registration and how does it improve research transparency?

Pre-registration is publicly documenting hypotheses, methods, and analysis plans before collecting/analyzing data. It distinguishes confirmatory from exploratory analyses and curbs p-hacking, HARKing, and selective reporting.

What is HARKing?

HARKing—Hypothesizing After the Results are Known—is presenting a post-hoc hypothesis (found by exploring the data) as if it had been predicted in advance. It inflates false positives and misrepresents exploratory findings as confirmatory.

What is publication bias and how does it distort the literature?

Publication bias is the tendency for statistically significant, positive results to be published while null or negative results go unpublished ('file drawer problem'). It causes the published literature to overstate effects and true effect sizes.

State the data-ink ratio principle of effective data visualization.

Tufte's principle: maximize the proportion of ink devoted to displaying actual data and minimize non-data ink (chartjunk, heavy gridlines, 3D effects, decoration). $$\text{data-ink ratio} = \frac{\text{data ink}}{\text{total ink}}$$

Name three ways graphics can mislead, and the fix for each.

(1) Truncated/non-zero y-axis exaggerating differences—use a full/appropriate baseline; (2) dual axes creating false correlations—avoid or label clearly; (3) area/volume scaling (bubbles) distorting magnitude—scale by area, not radius. Also: cherry-picked axis ranges, inconsistent scales, and misleading map binning.

When writing up statistical findings, what should be reported alongside a p-value, and how should uncertainty and effect magnitude be communicated?

Report the effect size with its confidence interval, the sample size, and the exact p-value (not just 'significant'). Describe the practical meaning, acknowledge limitations and assumptions, distinguish confirmatory from exploratory analyses, and avoid causal language for observational data.