Statistics & Probability Regression & Correlation Syllabus

Regression & Correlation syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Regression & Correlation in Statistics & Probability, not a summary of it.

1. 1

   Correlation Analysis

   4 topics

   • Pearson Correlation Coefficient
   • Coefficient of Determination
   • Correlation vs Causation
   • Rank Correlation
2. 2

   Simple Linear Regression

   4 topics

   • Least Squares Method
   • Interpreting Slope and Intercept
   • Residuals and Goodness of Fit
   • Inference on Regression Coefficients
3. 3

   Multiple Linear Regression

   4 topics

   • Model Specification
   • Multicollinearity
   • Model Selection
   • Interaction and Polynomial Terms
4. 4

   Regression Diagnostics

   4 topics

   • Linearity and Homoscedasticity
   • Normality of Residuals
   • Influential Points and Leverage
   • Heteroscedasticity Remedies
5. 5

   Generalized Linear Models

   3 topics

   • Logistic Regression
   • Poisson Regression
   • Link Functions

Regression & Correlation flashcards for Statistics & Probability

21 of 50 cards from the Regression & Correlation deck — real questions with worked answers.

1. Define the Pearson correlation coefficient $r$ and give its formula for a sample.

   $r$ measures the strength and direction of the *linear* association between two quantitative variables. $$r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^{2}}\,\sqrt{\sum (y_i - \bar{y})^{2}}} = \frac{S_{xy}}{\sqrt{S_{xx}\,S_{yy}}}$$

2. What is the range of the Pearson correlation coefficient $r$, and what do the extreme values mean?

   $-1 \leq r \leq 1$. Values of $r = +1$ or $r = -1$ indicate a perfect positive or negative linear relationship (all points lie exactly on a line); $r = 0$ indicates no linear relationship.

3. State three key properties of the Pearson correlation coefficient $r$.

   (1) It is dimensionless and unaffected by the units or by linear rescaling of $x$ or $y$. (2) It is symmetric: $r_{xy} = r_{yx}$. (3) It only measures *linear* association, so $r \approx 0$ can still occur with a strong nonlinear relationship.

4. How is the Pearson correlation $r$ related to the least-squares slope $b$ in simple linear regression?

   $$b = r\,\frac{s_y}{s_x}, \qquad r = b\,\frac{s_x}{s_y}$$ where $s_x, s_y$ are the sample standard deviations. Thus $r$ and $b$ always share the same sign.

5. Define the coefficient of determination $R^{2}$ and interpret it.

   $R^{2}$ is the proportion of the total variation in the response $y$ explained by the regression model: $$R^{2} = \frac{SS_{reg}}{SS_{tot}} = 1 - \frac{SS_{res}}{SS_{tot}}$$ It ranges from $0$ to $1$; e.g. $R^{2}=0.8$ means $80\%$ of the variance in $y$ is explained.

6. In simple linear regression, how does $R^{2}$ relate to the Pearson correlation $r$?

   $R^{2} = r^{2}$. Squaring the correlation coefficient gives the coefficient of determination (this identity holds only for simple linear regression with one predictor).

7. What is adjusted $R^{2}$ and why is it preferred for comparing models with different numbers of predictors?

   $$R^{2}_{adj} = 1 - \frac{SS_{res}/(n-p-1)}{SS_{tot}/(n-1)}$$ It penalizes adding predictors that do not improve fit. Unlike $R^{2}$, it can decrease when useless variables are added, making it fairer for model comparison ($n$ = sample size, $p$ = number of predictors).

8. Explain the principle 'correlation does not imply causation' and name a common cause of spurious correlation.

   A statistical association between $X$ and $Y$ does not establish that one causes the other. The link may be reversed, coincidental, or due to a **confounding (lurking) variable** $Z$ that influences both $X$ and $Y$ simultaneously.

9. What conditions are generally required to infer causation from a statistical relationship?

   (1) Association between the variables, (2) proper temporal order (cause precedes effect), and (3) elimination of alternative explanations/confounders — best achieved through a **randomized controlled experiment** rather than observational data.

10. Define Spearman's rank correlation coefficient $\rho$ and give its computational formula.

    Spearman's $\rho$ is Pearson's correlation applied to the ranks of the data. With distinct ranks and differences $d_i$ between paired ranks: $$\rho = 1 - \frac{6\sum d_i^{2}}{n(n^{2}-1)}$$ It measures the strength of a *monotonic* relationship.

11. When is a rank correlation (Spearman/Kendall) preferred over the Pearson correlation?

    When the relationship is monotonic but nonlinear, when the data are ordinal, or when there are outliers/non-normality. Rank methods are nonparametric and robust to outliers and monotonic transformations.

12. What does Kendall's $\tau$ measure, and how is it defined via concordant and discordant pairs?

    Kendall's $\tau$ measures rank agreement using pairs of observations: $$\tau = \frac{(\text{concordant pairs}) - (\text{discordant pairs})}{\binom{n}{2}}$$ A pair is concordant if both members are ordered the same way in $x$ and $y$; $\tau$ ranges from $-1$ to $1$.

13. State the objective of the least squares method in simple linear regression.

    It chooses the slope and intercept that minimize the sum of squared residuals: $$\min_{a,b} \sum_{i=1}^{n} \left(y_i - (a + b x_i)\right)^{2}$$ i.e. minimize the total squared vertical distance from points to the line.

14. Give the least-squares estimates for the slope $b$ and intercept $a$ in simple linear regression.

    $$b = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^{2}} = \frac{S_{xy}}{S_{xx}}, \qquad a = \bar{y} - b\,\bar{x}$$ The fitted line always passes through the point $(\bar{x}, \bar{y})$.

15. What are the normal equations for simple linear regression?

    $$\sum y_i = na + b\sum x_i, \qquad \sum x_i y_i = a\sum x_i + b\sum x_i^{2}$$ Solving these two simultaneous equations yields the least-squares estimates $a$ and $b$.

16. In the model $\hat{y} = a + bx$, how do you interpret the slope $b$?

    The slope $b$ is the estimated *change in the mean of $y$ for a one-unit increase in $x$*. Its sign gives the direction of the relationship and its magnitude the rate of change (in units of $y$ per unit of $x$).

17. In the model $\hat{y} = a + bx$, how do you interpret the intercept $a$, and when is it meaningful?

    The intercept $a$ is the predicted mean of $y$ when $x = 0$. It is only meaningful if $x = 0$ is within (or near) the observed range of the data; otherwise it is an extrapolation and may lack practical interpretation.

18. Define a residual $e_i$ in regression, and state a key property of the least-squares residuals.

    A residual is the difference between observed and fitted values: $e_i = y_i - \hat{y}_i$. For least-squares fits with an intercept, the residuals sum to zero, $\sum e_i = 0$, and are uncorrelated with the predictor.

19. Define the residual standard error (standard error of the estimate) $s_e$.

    $$s_e = \sqrt{\frac{\sum (y_i - \hat{y}_i)^{2}}{n - p - 1}} = \sqrt{\frac{SS_{res}}{n-p-1}}$$ It estimates the standard deviation of the residuals (typical prediction error) with $p$ predictors; for simple regression the denominator is $n-2$.

20. What is the decomposition of the total sum of squares in regression?

    $$SS_{tot} = SS_{reg} + SS_{res}$$ $$\sum (y_i - \bar{y})^{2} = \sum (\hat{y}_i - \bar{y})^{2} + \sum (y_i - \hat{y}_i)^{2}$$ Total variation = explained (regression) variation + unexplained (residual) variation.

21. What is the standard error of the slope estimate $b$ in simple linear regression?

    $$SE(b) = \frac{s_e}{\sqrt{\sum (x_i - \bar{x})^{2}}} = \frac{s_e}{\sqrt{S_{xx}}}$$ where $s_e$ is the residual standard error. It shrinks with larger spread in $x$ and larger sample size.

See more Regression & Correlation flashcards →

Planning Regression & Correlation for Statistics & Probability

Regression & Correlation (Statistics & Probability) FAQ