GATE Statistics Regression Analysis Syllabus

For any estimable linear combination $\vec{c}^{T}\vec{\beta}$, what does Gauss-Markov guarantee about $\vec{c}^{T}\hat{\vec{\beta}}$?

$\vec{c}^{T}\hat{\vec{\beta}}$ is the BLUE of $\vec{c}^{T}\vec{\beta}$: it is unbiased and has the minimum variance among all linear unbiased estimators of $\vec{c}^{T}\vec{\beta}$.

What happens to the Gauss-Markov result if errors are heteroscedastic or correlated, i.e. $\operatorname{Cov}(\vec{\varepsilon})=\sigma^{2}V$ with $V\neq I$?

OLS is still unbiased but no longer BLUE. The BLUE is the Generalized Least Squares (GLS) estimator $\hat{\vec{\beta}}_{GLS} = (X^{T}V^{-1}X)^{-1}X^{T}V^{-1}\vec{y}$ (Aitken's theorem).

What is an unbiased estimator of the error variance $\sigma^{2}$ in multiple regression?

$\hat{\sigma}^{2} = \frac{SSE}{n-p-1} = \frac{\vec{e}^{T}\vec{e}}{n-p-1}$, where $\vec{e}=\vec{y}-X\hat{\vec{\beta}}$ is the residual vector and $n-p-1$ is the residual degrees of freedom. This is the Mean Squared Error (MSE).

State the Fisher-Cochran Theorem (Cochran's Theorem).

Let $\vec{Z}\sim N(\vec{0},I_n)$ and suppose $\vec{Z}^{T}\vec{Z}=\sum_{i=1}^{n}Z_i^{2} = \sum_{j=1}^{k} Q_j$ where each $Q_j=\vec{Z}^{T}A_j\vec{Z}$ is a quadratic form with $\operatorname{rank}(A_j)=r_j$. If $\sum_{j=1}^{k} r_j = n$, then the $Q_j$ are independent and each $Q_j \sim \chi^{2}_{r_j}$.

In Cochran's Theorem, what condition on the ranks guarantees independence and chi-square distributions of the quadratic forms?

The ranks must add up to the total dimension: $\sum_{j} r_j = n$. Equivalently, the symmetric matrices $A_j$ must be idempotent and satisfy $\sum_j A_j = I_n$, which then forces $A_iA_j=0$ for $i\neq j$.

How does the Fisher-Cochran Theorem justify the independence of $SSR$ (or model SS) and $SSE$ in regression/ANOVA?

The total sum of squares decomposes into orthogonal idempotent quadratic forms whose ranks sum to $n$. Cochran's Theorem then implies the regression and error sums of squares are independent and each is $\sigma^{2}$ times a chi-square variable, enabling the $F$-test.

Under normal errors, what is the distribution of $\frac{SSE}{\sigma^{2}}$ in multiple regression, and why (via Cochran)?

$\frac{SSE}{\sigma^{2}} \sim \chi^{2}_{n-p-1}$. The residual quadratic form $\vec{\varepsilon}^{T}(I-H)\vec{\varepsilon}/\sigma^{2}$ uses the idempotent matrix $I-H$ of rank $n-p-1$, so by Cochran it is chi-square with $n-p-1$ d.f., independent of $\hat{\vec{\beta}}$.

What is the hat matrix $H$ in regression, and what are its key properties used with Cochran's Theorem?

$H = X(X^{T}X)^{-1}X^{T}$, giving $\hat{\vec{y}}=H\vec{y}$. It is symmetric and idempotent ($H^{2}=H$, $H^{T}=H$) with $\operatorname{rank}(H)=\operatorname{trace}(H)=p+1$. Then $I-H$ is also idempotent with rank $n-p-1$.

A consequence of Cochran/normality: what is the joint distribution structure of $\hat{\vec{\beta}}$ and $SSE$?

$\hat{\vec{\beta}} \sim N(\vec{\beta},\,\sigma^{2}(X^{T}X)^{-1})$ and $SSE/\sigma^{2}\sim\chi^{2}_{n-p-1}$, and the two are statistically independent. This independence is what makes the studentized statistics follow a $t$ distribution.

Using normality, what is the distribution of $\frac{\hat{\beta}_j - \beta_j}{\operatorname{se}(\hat{\beta}_j)}$, and what is $\operatorname{se}(\hat{\beta}_j)$?

It follows a $t$ distribution with $n-p-1$ degrees of freedom: $\frac{\hat{\beta}_j-\beta_j}{\operatorname{se}(\hat{\beta}_j)} \sim t_{n-p-1}$, where $\operatorname{se}(\hat{\beta}_j)=\hat{\sigma}\sqrt{(X^{T}X)^{-1}_{jj}}$ and $\hat{\sigma}=\sqrt{MSE}$.

Give the formula for a $100(1-\alpha)\%$ confidence interval for a regression coefficient $\beta_j$.

$$\hat{\beta}_j \pm t_{\alpha/2,\,n-p-1}\cdot \operatorname{se}(\hat{\beta}_j),\qquad \operatorname{se}(\hat{\beta}_j)=\hat{\sigma}\sqrt{(X^{T}X)^{-1}_{jj}}.$$

In simple linear regression $y=\beta_0+\beta_1 x+\varepsilon$, give the $100(1-\alpha)\%$ confidence interval for the slope $\beta_1$.

$$\hat{\beta}_1 \pm t_{\alpha/2,\,n-2}\cdot \frac{\hat{\sigma}}{\sqrt{S_{xx}}},\qquad S_{xx}=\sum_{i}(x_i-\bar{x})^{2},\ \hat{\sigma}=\sqrt{\tfrac{SSE}{n-2}}.$$

Why do confidence intervals for regression coefficients use the $t$ distribution rather than the standard normal $z$?

Because the error variance $\sigma^{2}$ is unknown and estimated by $\hat{\sigma}^{2}=MSE$. Replacing $\sigma$ by $\hat{\sigma}$ (a chi-square-based estimate) makes the studentized statistic follow $t_{n-p-1}$ rather than $N(0,1)$.

What is the formula for a confidence interval for the MEAN response $E[y|x_0]$ at a point $\vec{x}_0$ in multiple regression?

$$\vec{x}_0^{T}\hat{\vec{\beta}} \pm t_{\alpha/2,\,n-p-1}\,\hat{\sigma}\sqrt{\vec{x}_0^{T}(X^{T}X)^{-1}\vec{x}_0}.$$ It estimates the average response at $\vec{x}_0$.

What is the formula for a PREDICTION interval for a new individual observation $y_0$ at $\vec{x}_0$?

$$\vec{x}_0^{T}\hat{\vec{\beta}} \pm t_{\alpha/2,\,n-p-1}\,\hat{\sigma}\sqrt{1+\vec{x}_0^{T}(X^{T}X)^{-1}\vec{x}_0}.$$ The extra $+1$ accounts for the variability of the new error term.

Compare a confidence interval for the mean response with a prediction interval for a new observation at the same $\vec{x}_0$. Which is wider and why?

The prediction interval is wider. It includes both the uncertainty in estimating the mean ($\vec{x}_0^{T}(X^{T}X)^{-1}\vec{x}_0$) and the irreducible error variance of a new point (the extra $+1$ under the root), whereas the confidence interval includes only the former.

For the simple-regression mean response at $x_0$, what is the standard error term, and where is it smallest?

$\operatorname{se}(\hat{y}_0)=\hat{\sigma}\sqrt{\frac{1}{n}+\frac{(x_0-\bar{x})^{2}}{S_{xx}}}$. It is smallest at $x_0=\bar{x}$ and grows as $x_0$ moves away from $\bar{x}$, so the confidence band is narrowest at the mean of $x$.

How is the width of a confidence interval for $\beta_j$ affected by sample size $n$ and the spread of the predictor?

Width $\propto \hat{\sigma}\sqrt{(X^{T}X)^{-1}_{jj}}$. Larger $n$ and greater predictor spread ($S_{xx}$) shrink $\operatorname{se}(\hat{\beta}_j)$ and narrow the interval; higher residual variance $\hat{\sigma}^{2}$ or multicollinearity widens it.

What is the relationship between the $100(1-\alpha)\%$ confidence interval for $\beta_j$ and the two-sided $t$-test of $H_0:\beta_j=0$?

They are equivalent (duality): the interval excludes $0$ if and only if $H_0:\beta_j=0$ is rejected at significance level $\alpha$. The CI is the set of null values not rejected by the test.

State the overall $F$-test of regression significance and the distribution of its statistic under $H_0$.

$H_0:\beta_1=\cdots=\beta_p=0$. Statistic $F=\frac{SSR/p}{SSE/(n-p-1)}=\frac{MSR}{MSE}$. Under $H_0$ (and normality), $F\sim F_{p,\,n-p-1}$. Cochran's Theorem guarantees the independence of $MSR$ and $MSE$.

Express the overall $F$-statistic in terms of $R^{2}$.

$$F = \frac{R^{2}/p}{(1-R^{2})/(n-p-1)} = \frac{R^{2}}{1-R^{2}}\cdot\frac{n-p-1}{p}.$$ It is a monotonic increasing function of $R^{2}$.

What is a confidence interval for the error variance $\sigma^{2}$ in regression, and which theorem justifies its chi-square basis?

$$\left(\frac{(n-p-1)\hat{\sigma}^{2}}{\chi^{2}_{\alpha/2,\,n-p-1}},\ \frac{(n-p-1)\hat{\sigma}^{2}}{\chi^{2}_{1-\alpha/2,\,n-p-1}}\right).$$ It relies on $\frac{(n-p-1)\hat{\sigma}^{2}}{\sigma^{2}}\sim\chi^{2}_{n-p-1}$, a consequence of Cochran's Theorem.

List four common applications of regression analysis.

(1) Prediction/forecasting of a response from predictors; (2) Quantifying and testing the strength of relationships between variables; (3) Estimating effect sizes / parameter inference (e.g. marginal effect of $x_j$); (4) Control and optimization of a process by setting predictor levels.

How is the slope coefficient $\beta_j$ interpreted in a multiple regression application?

$\beta_j$ is the expected change in the response $y$ for a one-unit increase in $x_j$, holding all other predictors fixed (the ceteris paribus / partial effect).

In applications, what distinguishes interpolation from extrapolation in regression prediction, and why is extrapolation risky?

Interpolation predicts within the observed range of the predictors; extrapolation predicts outside it. Extrapolation is risky because the fitted relationship may not hold beyond the data range, and prediction intervals widen rapidly as $\vec{x}_0$ leaves the data cloud.

What is multicollinearity, and how does it affect a regression application?

Multicollinearity is strong linear dependence among predictors, making $X^{T}X$ near-singular. It inflates the variances $\sigma^{2}(X^{T}X)^{-1}_{jj}$ of coefficient estimates, widening confidence intervals and making individual coefficients unstable/insignificant, though predictions may still be fine.

Compare $R^{2}$, Adjusted $R^{2}$, and Mallows' $C_p$ as criteria for model selection in applications.

$R^{2}$ always favors more predictors and cannot select. Adjusted $R^{2}$ penalizes complexity via degrees of freedom and can decrease. Mallows' $C_p=\frac{SSE_p}{\hat{\sigma}^{2}}-(n-2p)$ targets prediction error; a good model has $C_p\approx p$. Adjusted $R^{2}$ and $C_p$ both balance fit vs. complexity, unlike raw $R^{2}$.

What is the purpose of residual analysis in a regression application?

Residual plots check model assumptions: a random scatter of $e_i$ vs. $\hat{y}_i$ supports linearity and homoscedasticity; patterns/funnels indicate misspecification or heteroscedasticity; a normal Q-Q plot of residuals checks the normality assumption needed for $t$/$F$ inference.

When the linear model assumptions hold with normal errors, what does combining Gauss-Markov and Cochran's results tell us about the optimality of OLS?

Gauss-Markov makes OLS the BLUE without normality. Adding normality, Cochran's Theorem gives independent chi-square sums of squares, and OLS coincides with the MLE and becomes the Minimum Variance Unbiased Estimator (MVUE) among ALL unbiased estimators, not just linear ones, attaining the Cramer-Rao bound.