Statistics & Probability Regression & Correlation Syllabus

What is the standard error of the slope estimate $b$ in simple linear regression?

$$SE(b) = \frac{s_e}{\sqrt{\sum (x_i - \bar{x})^{2}}} = \frac{s_e}{\sqrt{S_{xx}}}$$ where $s_e$ is the residual standard error. It shrinks with larger spread in $x$ and larger sample size.

State the $t$-test used to test whether a regression coefficient $\beta_j$ is significant.

To test $H_0: \beta_j = 0$ vs $H_1: \beta_j \neq 0$, use $$t = \frac{\hat{\beta}_j}{SE(\hat{\beta}_j)}$$ compared to a $t$-distribution with $n - p - 1$ degrees of freedom. Rejecting $H_0$ indicates $x_j$ is a significant predictor.

How is a $(1-\alpha)$ confidence interval for a regression coefficient $\beta_j$ constructed?

$$\hat{\beta}_j \pm t_{\alpha/2,\,n-p-1}\cdot SE(\hat{\beta}_j)$$ If this interval excludes $0$, the coefficient is statistically significant at level $\alpha$.

What does the overall $F$-test in multiple regression assess, and what is its statistic?

It tests whether *at least one* predictor has nonzero coefficient ($H_0: \beta_1 = \dots = \beta_p = 0$). $$F = \frac{SS_{reg}/p}{SS_{res}/(n-p-1)} = \frac{MS_{reg}}{MS_{res}}$$ with $(p,\, n-p-1)$ degrees of freedom.

What is meant by 'model specification' and what are the consequences of omitting a relevant variable?

Model specification is choosing the correct functional form and set of predictors. **Omitting a relevant variable** that is correlated with included predictors causes **omitted-variable bias**: the estimated coefficients become biased and inconsistent.

What is the consequence of including an irrelevant variable in a regression model?

Including an irrelevant predictor (true coefficient $0$) keeps the coefficient estimates unbiased but **increases their variance/standard errors**, reducing efficiency and precision of the estimates and predictions.

Define multicollinearity and describe its effect on a regression model.

Multicollinearity is high linear correlation among two or more predictors. It inflates the standard errors of coefficients, making them unstable and hard to interpret (large swings, wrong signs), though it does not reduce the model's overall predictive power or $R^{2}$.

Define the Variance Inflation Factor (VIF) and state a common rule of thumb for problematic multicollinearity.

$$VIF_j = \frac{1}{1 - R_j^{2}}$$ where $R_j^{2}$ is from regressing predictor $x_j$ on all other predictors. A common rule: $VIF > 5$ (or $> 10$) signals serious multicollinearity.

List three ways to remedy multicollinearity in a regression model.

(1) Remove or combine redundant/highly correlated predictors; (2) collect more data or use principal component / partial least squares regression; (3) apply penalized regression such as **ridge regression**, which shrinks coefficients and stabilizes estimates.

Contrast forward selection, backward elimination, and stepwise selection.

**Forward selection** starts empty and adds the most significant predictor at each step. **Backward elimination** starts with all predictors and removes the least significant. **Stepwise** combines both, adding and removing variables at each step based on entry/exit criteria.

Define AIC and BIC and state how they are used in model selection.

For a fitted model with $k$ parameters and maximized likelihood $\hat{L}$: $$AIC = 2k - 2\ln \hat{L}, \qquad BIC = k\ln n - 2\ln \hat{L}$$ Both trade off fit against complexity; the model with the **lowest** value is preferred. BIC penalizes complexity more heavily for large $n$.

Why should you generally include lower-order terms when a model contains an interaction or polynomial term (principle of hierarchy)?

By the **hierarchy principle**, if a model includes $x_1 x_2$ it should also include $x_1$ and $x_2$ (and if it includes $x^{2}$, also $x$). This keeps the model invariant to shifts of the variables and makes coefficients interpretable.

In the model $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_1 x_2$, what does the interaction coefficient $\beta_3$ represent?

$\beta_3$ measures how the effect of $x_1$ on $y$ changes as $x_2$ changes (and vice versa). The slope of $y$ with respect to $x_1$ is $\beta_1 + \beta_3 x_2$, so it depends on the level of $x_2$.

Why is fitting $y = \beta_0 + \beta_1 x + \beta_2 x^{2}$ still called 'linear regression'?

Because 'linear' refers to linearity in the *parameters* $\beta_j$, not in the predictors. A polynomial model is linear in $\beta_0, \beta_1, \beta_2$ and can be fit with ordinary least squares by treating $x^{2}$ as an additional predictor.

State the key assumptions of the classical linear regression model.

(1) **Linearity** of $E[y|x]$ in the parameters; (2) **independence** of errors; (3) **homoscedasticity** (constant error variance); (4) **normality** of errors; and predictors measured without error / no perfect multicollinearity. Concisely: errors are i.i.d. $N(0, \sigma^{2})$.

Define homoscedasticity and heteroscedasticity.

**Homoscedasticity**: the error variance is constant across all levels of the predictors, $\operatorname{Var}(\varepsilon_i) = \sigma^{2}$. **Heteroscedasticity**: the error variance changes with $x$ (e.g. spreads out), often seen as a funnel/fan shape in a residuals-vs-fitted plot.

Which diagnostic plot best checks the linearity and homoscedasticity assumptions, and what pattern indicates a violation?

A **residuals-vs-fitted-values plot**. Ideally residuals scatter randomly around $0$ with constant spread. A curved pattern signals nonlinearity; a funnel/fan shape (spread changing with fitted value) signals heteroscedasticity.

How can the normality-of-residuals assumption be checked, and why does it matter?

Check with a **normal Q-Q plot** of residuals (points should lie on the diagonal) or a formal test (Shapiro-Wilk). Normality mainly matters for the validity of $t$- and $F$-based inference and confidence/prediction intervals in small samples; by the CLT it is less critical for large $n$.

Define leverage in regression and give the average leverage value.

Leverage $h_{ii}$ (the $i$-th diagonal of the hat matrix $H = X(X^{T}X)^{-1}X^{T}$) measures how far an observation's predictor values are from the mean, i.e. its potential to influence its own fitted value. Average leverage is $(p+1)/n$; values exceeding $2(p+1)/n$ are considered high.

Distinguish an outlier, a high-leverage point, and an influential point.

An **outlier** has a large residual (unusual $y$). A **high-leverage** point has unusual predictor values (unusual $x$). An **influential** point substantially changes the fitted model when removed — typically a point that is both an outlier *and* high leverage.

Define Cook's distance and its purpose.

$$D_i = \frac{e_i^{2}}{p\,s_e^{2}}\cdot\frac{h_{ii}}{(1-h_{ii})^{2}}$$ Cook's distance measures the overall influence of observation $i$ on all fitted values by combining its residual and leverage. Points with $D_i > 1$ (or $> 4/n$) are flagged as influential.

Name three remedies for heteroscedasticity.

(1) **Transform the response** (e.g. log or square-root of $y$) to stabilize variance; (2) use **weighted least squares (WLS)** with weights inversely proportional to the variance; (3) use **heteroscedasticity-robust (White/sandwich) standard errors** for valid inference.

What is weighted least squares (WLS) and when is it used?

WLS minimizes $\sum w_i (y_i - \hat{y}_i)^{2}$, weighting observations by $w_i = 1/\sigma_i^{2}$. It is used under heteroscedasticity to give less weight to high-variance observations, restoring efficient, minimum-variance estimates.

Write the logistic regression model for a binary outcome and identify the link function.

$$p = P(y=1\mid x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x)}}$$ Equivalently the **logit link**: $$\ln\!\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 x$$ modeling the log-odds as a linear function of the predictors.

How is a logistic-regression coefficient $\beta_j$ interpreted in terms of odds?

$e^{\beta_j}$ is the **odds ratio**: a one-unit increase in $x_j$ multiplies the odds of the outcome by $e^{\beta_j}$ (holding other predictors fixed). $\beta_j > 0$ raises the odds, $\beta_j < 0$ lowers them, $\beta_j = 0$ means no effect.

How are logistic regression coefficients estimated, and why not ordinary least squares?

By **maximum likelihood estimation (MLE)**, maximizing the Bernoulli likelihood $\prod p_i^{y_i}(1-p_i)^{1-y_i}$ (solved iteratively, e.g. IRLS). OLS is unsuitable because the response is binary and the errors are non-normal with non-constant variance.

Write the Poisson regression model and state the type of response it is used for.

For count responses $y \in \{0,1,2,\dots\}$, Poisson regression uses a **log link**: $$\ln(\lambda) = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p, \qquad \lambda = E[y\mid x]$$ so $\lambda = e^{\beta_0 + \sum \beta_j x_j} > 0$.

How is a Poisson-regression coefficient $\beta_j$ interpreted?

$e^{\beta_j}$ is the **rate ratio (incidence-rate ratio)**: a one-unit increase in $x_j$ multiplies the expected count $\lambda$ by $e^{\beta_j}$, holding other predictors constant.

What key assumption does the Poisson model make about the response, and what is overdispersion?

Poisson assumes the conditional **mean equals the variance**: $E[y] = \operatorname{Var}(y) = \lambda$. **Overdispersion** occurs when the observed variance exceeds the mean; it is handled with quasi-Poisson or **negative binomial regression**.

Compare linear, logistic, and Poisson regression by response type and link function.

**Linear**: continuous response, identity link, $E[y]=\beta_0+\beta_1 x$. **Logistic**: binary response, logit link, models log-odds. **Poisson**: count response, log link, models log of expected count. All three are generalized linear models differing in the distribution and link.