Statistics & Probability Study Design & Sampling Syllabus

State the design principle summarized as 'block what you can, randomize what you cannot.'

Use blocking to control the effect of variables you can identify and measure; use randomization to average out the effect of variables you cannot identify or control.

In a factorial design, what does a $2 \times 3$ design mean and how many treatment combinations does it have?

Two factors, one with $2$ levels and one with $3$ levels; it has $2 \times 3 = 6$ treatment combinations.

What key advantage do factorial designs offer over one-factor-at-a-time experiments?

They allow estimation of interaction effects between factors (and efficiently estimate main effects) using the same experimental units.

Define a confounding variable.

A variable associated with both the explanatory variable and the response variable, so that its effect cannot be separated from the effect of the explanatory variable.

Why does confounding prevent causal conclusions in observational studies?

Because any observed association between explanatory and response variables may be due to the confounder, so cause and effect cannot be distinguished from mere association.

What distinguishes an observational study from a designed experiment?

In an observational study researchers only measure/observe variables without assigning treatments; in an experiment researchers deliberately impose a treatment on subjects.

What is a cohort study?

A prospective (usually forward-looking) observational study that follows a group sharing a characteristic or exposure over time to see who develops the outcome of interest.

What is a case-control study?

A retrospective observational study that starts with subjects who have the outcome (cases) and those who do not (controls), then looks backward to compare prior exposures.

Give one advantage of a case-control study over a cohort study.

It is efficient for rare outcomes/diseases and is faster and cheaper because it does not require long follow-up.

Which measure of association is typically reported by a case-control study, and why not risk directly?

The odds ratio; because subjects are selected by outcome status, incidence/risk cannot be estimated directly from the design.

What is a cross-sectional study?

An observational study that measures exposure and outcome simultaneously in a population at a single point in time (a 'snapshot').

State one key limitation of cross-sectional studies regarding causation.

Because exposure and outcome are measured at the same time, temporal order cannot be established, so they cannot demonstrate that exposure preceded (caused) the outcome.

Name three common sources of bias in observational research.

Selection bias, confounding, and information/measurement bias (e.g., recall or observer bias).

What is selection bias?

Systematic error arising when the individuals included in a study differ systematically from the target population, so the sample is unrepresentative.

Define sampling error.

The natural variability between a sample statistic and the true population parameter that arises purely because only a subset of the population is observed; it decreases as sample size increases.

Define non-sampling error and give an example.

Any error not due to sampling variability, such as measurement error, non-response, coverage/undercoverage, or data-processing mistakes; e.g., a poorly calibrated instrument. It is not reduced by increasing sample size.

How does increasing sample size $n$ affect sampling error versus non-sampling error?

It reduces sampling error (standard error shrinks roughly like $\frac{1}{\sqrt{n}}$) but does not reduce non-sampling error, which may even grow.

What is response bias in a survey?

Systematic error caused by respondents giving inaccurate answers, e.g., due to social desirability, question wording, or interviewer influence.

What is non-response bias?

Bias that occurs when individuals selected for the sample do not respond and those non-respondents differ systematically from respondents.

How does the wording or order of survey questions affect results?

Leading, loaded, or confusingly worded questions and question order can systematically push respondents toward certain answers, producing wording/order bias.

What is undercoverage (coverage bias) in a survey?

When some groups of the target population are inadequately represented or excluded from the sampling frame, so they have little or no chance of selection.

What is survivorship bias?

A logical error of focusing only on the subjects/objects that 'survived' a selection process while overlooking those that did not, leading to overly optimistic or distorted conclusions.

Give a classic illustration of survivorship bias (WWII aircraft).

Reinforcing bombers where returning planes showed bullet holes was wrong; the planes hit in fatal areas never returned, so armor was actually needed where the surviving planes showed no damage.

Distinguish replication within an experiment from replication of an experiment.

Within-experiment replication means applying each treatment to multiple experimental units to estimate variability; replication of an experiment means independent researchers repeating the whole study to confirm findings.

Distinguish reproducibility from replicability in scientific research.

Reproducibility: obtaining the same results using the original author's same data and analysis. Replicability: obtaining consistent results in a new study that collects new data to answer the same question.

Why is replication important for scientific validity?

Consistent results across independent repetitions provide evidence that a finding is real and not due to chance, error, or study-specific artifacts.

What is quota sampling and how does it differ from stratified sampling?

Quota sampling fills preset counts within subgroups using non-random (e.g., convenience) selection; stratified sampling uses random selection within each stratum, making it a probability method.

What is the difference between the sampling frame and the target population?

The target population is everyone you want to draw conclusions about; the sampling frame is the actual list from which the sample is drawn. Mismatches between them cause coverage error.

Why does randomization support cause-and-effect conclusions while observational studies generally cannot?

Random assignment makes treatment groups probabilistically equivalent on all confounders, so a difference in outcomes can be causally attributed to the treatment; observational studies lack this control, leaving confounding as an alternative explanation.

What is recall bias and in which study design is it most problematic?

A form of information bias where subjects remember past exposures inaccurately; it is most problematic in retrospective case-control studies because cases and controls may recall exposures differently.