GATE Statistics Regression Analysis Flashcards

24 sample cards from the Regression Analysis deck

Sampled from the end of the deck, so these are different cards from the ones shown on the syllabus page.

1. How does the Fisher-Cochran Theorem justify the independence of $SSR$ (or model SS) and $SSE$ in regression/ANOVA?

   The total sum of squares decomposes into orthogonal idempotent quadratic forms whose ranks sum to $n$. Cochran's Theorem then implies the regression and error sums of squares are independent and each is $\sigma^{2}$ times a chi-square variable, enabling the $F$-test.

2. Under normal errors, what is the distribution of $\frac{SSE}{\sigma^{2}}$ in multiple regression, and why (via Cochran)?

   $\frac{SSE}{\sigma^{2}} \sim \chi^{2}_{n-p-1}$. The residual quadratic form $\vec{\varepsilon}^{T}(I-H)\vec{\varepsilon}/\sigma^{2}$ uses the idempotent matrix $I-H$ of rank $n-p-1$, so by Cochran it is chi-square with $n-p-1$ d.f., independent of $\hat{\vec{\beta}}$.

3. What is the hat matrix $H$ in regression, and what are its key properties used with Cochran's Theorem?

   $H = X(X^{T}X)^{-1}X^{T}$, giving $\hat{\vec{y}}=H\vec{y}$. It is symmetric and idempotent ($H^{2}=H$, $H^{T}=H$) with $\operatorname{rank}(H)=\operatorname{trace}(H)=p+1$. Then $I-H$ is also idempotent with rank $n-p-1$.

4. A consequence of Cochran/normality: what is the joint distribution structure of $\hat{\vec{\beta}}$ and $SSE$?

   $\hat{\vec{\beta}} \sim N(\vec{\beta},\,\sigma^{2}(X^{T}X)^{-1})$ and $SSE/\sigma^{2}\sim\chi^{2}_{n-p-1}$, and the two are statistically independent. This independence is what makes the studentized statistics follow a $t$ distribution.

5. Using normality, what is the distribution of $\frac{\hat{\beta}_j - \beta_j}{\operatorname{se}(\hat{\beta}_j)}$, and what is $\operatorname{se}(\hat{\beta}_j)$?

   It follows a $t$ distribution with $n-p-1$ degrees of freedom: $\frac{\hat{\beta}_j-\beta_j}{\operatorname{se}(\hat{\beta}_j)} \sim t_{n-p-1}$, where $\operatorname{se}(\hat{\beta}_j)=\hat{\sigma}\sqrt{(X^{T}X)^{-1}_{jj}}$ and $\hat{\sigma}=\sqrt{MSE}$.

6. Give the formula for a $100(1-\alpha)\%$ confidence interval for a regression coefficient $\beta_j$.

   $$\hat{\beta}_j \pm t_{\alpha/2,\,n-p-1}\cdot \operatorname{se}(\hat{\beta}_j),\qquad \operatorname{se}(\hat{\beta}_j)=\hat{\sigma}\sqrt{(X^{T}X)^{-1}_{jj}}.$$

7. In simple linear regression $y=\beta_0+\beta_1 x+\varepsilon$, give the $100(1-\alpha)\%$ confidence interval for the slope $\beta_1$.

   $$\hat{\beta}_1 \pm t_{\alpha/2,\,n-2}\cdot \frac{\hat{\sigma}}{\sqrt{S_{xx}}},\qquad S_{xx}=\sum_{i}(x_i-\bar{x})^{2},\ \hat{\sigma}=\sqrt{\tfrac{SSE}{n-2}}.$$

8. Why do confidence intervals for regression coefficients use the $t$ distribution rather than the standard normal $z$?

   Because the error variance $\sigma^{2}$ is unknown and estimated by $\hat{\sigma}^{2}=MSE$. Replacing $\sigma$ by $\hat{\sigma}$ (a chi-square-based estimate) makes the studentized statistic follow $t_{n-p-1}$ rather than $N(0,1)$.

9. What is the formula for a confidence interval for the MEAN response $E[y|x_0]$ at a point $\vec{x}_0$ in multiple regression?

   $$\vec{x}_0^{T}\hat{\vec{\beta}} \pm t_{\alpha/2,\,n-p-1}\,\hat{\sigma}\sqrt{\vec{x}_0^{T}(X^{T}X)^{-1}\vec{x}_0}.$$ It estimates the average response at $\vec{x}_0$.

10. What is the formula for a PREDICTION interval for a new individual observation $y_0$ at $\vec{x}_0$?

    $$\vec{x}_0^{T}\hat{\vec{\beta}} \pm t_{\alpha/2,\,n-p-1}\,\hat{\sigma}\sqrt{1+\vec{x}_0^{T}(X^{T}X)^{-1}\vec{x}_0}.$$ The extra $+1$ accounts for the variability of the new error term.

11. Compare a confidence interval for the mean response with a prediction interval for a new observation at the same $\vec{x}_0$. Which is wider and why?

    The prediction interval is wider. It includes both the uncertainty in estimating the mean ($\vec{x}_0^{T}(X^{T}X)^{-1}\vec{x}_0$) and the irreducible error variance of a new point (the extra $+1$ under the root), whereas the confidence interval includes only the former.

12. For the simple-regression mean response at $x_0$, what is the standard error term, and where is it smallest?

    $\operatorname{se}(\hat{y}_0)=\hat{\sigma}\sqrt{\frac{1}{n}+\frac{(x_0-\bar{x})^{2}}{S_{xx}}}$. It is smallest at $x_0=\bar{x}$ and grows as $x_0$ moves away from $\bar{x}$, so the confidence band is narrowest at the mean of $x$.

13. How is the width of a confidence interval for $\beta_j$ affected by sample size $n$ and the spread of the predictor?

    Width $\propto \hat{\sigma}\sqrt{(X^{T}X)^{-1}_{jj}}$. Larger $n$ and greater predictor spread ($S_{xx}$) shrink $\operatorname{se}(\hat{\beta}_j)$ and narrow the interval; higher residual variance $\hat{\sigma}^{2}$ or multicollinearity widens it.

14. What is the relationship between the $100(1-\alpha)\%$ confidence interval for $\beta_j$ and the two-sided $t$-test of $H_0:\beta_j=0$?

    They are equivalent (duality): the interval excludes $0$ if and only if $H_0:\beta_j=0$ is rejected at significance level $\alpha$. The CI is the set of null values not rejected by the test.

15. State the overall $F$-test of regression significance and the distribution of its statistic under $H_0$.

    $H_0:\beta_1=\cdots=\beta_p=0$. Statistic $F=\frac{SSR/p}{SSE/(n-p-1)}=\frac{MSR}{MSE}$. Under $H_0$ (and normality), $F\sim F_{p,\,n-p-1}$. Cochran's Theorem guarantees the independence of $MSR$ and $MSE$.

16. Express the overall $F$-statistic in terms of $R^{2}$.

    $$F = \frac{R^{2}/p}{(1-R^{2})/(n-p-1)} = \frac{R^{2}}{1-R^{2}}\cdot\frac{n-p-1}{p}.$$ It is a monotonic increasing function of $R^{2}$.

17. What is a confidence interval for the error variance $\sigma^{2}$ in regression, and which theorem justifies its chi-square basis?

    $$\left(\frac{(n-p-1)\hat{\sigma}^{2}}{\chi^{2}_{\alpha/2,\,n-p-1}},\ \frac{(n-p-1)\hat{\sigma}^{2}}{\chi^{2}_{1-\alpha/2,\,n-p-1}}\right).$$ It relies on $\frac{(n-p-1)\hat{\sigma}^{2}}{\sigma^{2}}\sim\chi^{2}_{n-p-1}$, a consequence of Cochran's Theorem.

18. List four common applications of regression analysis.

    (1) Prediction/forecasting of a response from predictors; (2) Quantifying and testing the strength of relationships between variables; (3) Estimating effect sizes / parameter inference (e.g. marginal effect of $x_j$); (4) Control and optimization of a process by setting predictor levels.

19. How is the slope coefficient $\beta_j$ interpreted in a multiple regression application?

    $\beta_j$ is the expected change in the response $y$ for a one-unit increase in $x_j$, holding all other predictors fixed (the ceteris paribus / partial effect).

20. In applications, what distinguishes interpolation from extrapolation in regression prediction, and why is extrapolation risky?

    Interpolation predicts within the observed range of the predictors; extrapolation predicts outside it. Extrapolation is risky because the fitted relationship may not hold beyond the data range, and prediction intervals widen rapidly as $\vec{x}_0$ leaves the data cloud.

21. What is multicollinearity, and how does it affect a regression application?

    Multicollinearity is strong linear dependence among predictors, making $X^{T}X$ near-singular. It inflates the variances $\sigma^{2}(X^{T}X)^{-1}_{jj}$ of coefficient estimates, widening confidence intervals and making individual coefficients unstable/insignificant, though predictions may still be fine.

22. Compare $R^{2}$, Adjusted $R^{2}$, and Mallows' $C_p$ as criteria for model selection in applications.

    $R^{2}$ always favors more predictors and cannot select. Adjusted $R^{2}$ penalizes complexity via degrees of freedom and can decrease. Mallows' $C_p=\frac{SSE_p}{\hat{\sigma}^{2}}-(n-2p)$ targets prediction error; a good model has $C_p\approx p$. Adjusted $R^{2}$ and $C_p$ both balance fit vs. complexity, unlike raw $R^{2}$.

23. What is the purpose of residual analysis in a regression application?

    Residual plots check model assumptions: a random scatter of $e_i$ vs. $\hat{y}_i$ supports linearity and homoscedasticity; patterns/funnels indicate misspecification or heteroscedasticity; a normal Q-Q plot of residuals checks the normality assumption needed for $t$/$F$ inference.

24. When the linear model assumptions hold with normal errors, what does combining Gauss-Markov and Cochran's results tell us about the optimality of OLS?

    Gauss-Markov makes OLS the BLUE without normality. Adding normality, Cochran's Theorem gives independent chi-square sums of squares, and OLS coincides with the MLE and becomes the Minimum Variance Unbiased Estimator (MVUE) among ALL unbiased estimators, not just linear ones, attaining the Cramer-Rao bound.

What this deck covers

Regression Analysis flashcards FAQ