GATE Statistics Regression Analysis Syllabus

Regression Analysis syllabus — full chapter and topic list

Expand any chapter to see its topics and sub-topics. This is the whole examinable outline for Regression Analysis in GATE Statistics, not a summary of it.

1. 1

   Simple and Multiple Linear Regression

   2 topics

   • R2 and Adjusted R2
   • Applications
2. 2

   Distributions of Quadratic Forms of Random Vectors

   2 topics

   • Fisher-Cochran Theorem
   • Gauss-Markov Theorem
3. 3

   Tests for Regression Coefficients

   1 topic

   • Confidence Intervals

Regression Analysis flashcards for GATE Statistics

25 of 49 cards from the Regression Analysis deck — real questions with worked answers.

1. In simple/multiple linear regression, what is the definition of the coefficient of determination $R^{2}$ in terms of the total and residual sums of squares?

   $R^{2} = \frac{SSR}{SST} = 1 - \frac{SSE}{SST}$, where $SST = \sum (y_i - \bar{y})^{2}$, $SSR = \sum (\hat{y}_i - \bar{y})^{2}$, and $SSE = \sum (y_i - \hat{y}_i)^{2}$.

2. What is the fundamental sum-of-squares identity (ANOVA decomposition) underlying $R^{2}$?

   $$SST = SSR + SSE,\quad \text{i.e.}\quad \sum_{i}(y_i-\bar{y})^{2} = \sum_{i}(\hat{y}_i-\bar{y})^{2} + \sum_{i}(y_i-\hat{y}_i)^{2}.$$ This holds when the model includes an intercept.

3. What is the range of $R^{2}$, and what do the extreme values mean?

   $0 \leq R^{2} \leq 1$. $R^{2}=1$ means the model explains all variation (perfect fit, $SSE=0$); $R^{2}=0$ means the model explains none of the variation beyond the mean ($SSR=0$).

4. How is $R^{2}$ interpreted in words?

   $R^{2}$ is the proportion of the total variability in the response $y$ that is explained by the fitted regression model (the predictors).

5. In simple linear regression with one predictor, how is $R^{2}$ related to the Pearson correlation coefficient $r$ between $x$ and $y$?

   $R^{2} = r^{2}$, where $r = \frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum (x_i-\bar{x})^{2}}\,\sqrt{\sum (y_i-\bar{y})^{2}}}$. The sign of $r$ matches the sign of the slope.

6. Why can $R^{2}$ be misleading as a model-selection criterion when comparing models with different numbers of predictors?

   $R^{2}$ never decreases when a predictor is added (it is non-decreasing in the number of regressors), even if the new predictor is irrelevant. So it always favors larger models and cannot penalize complexity.

7. Give the formula for the Adjusted $R^{2}$ ($\bar{R}^{2}$) for a model with $n$ observations and $p$ predictors (excluding intercept).

   $$\bar{R}^{2} = 1 - \frac{SSE/(n-p-1)}{SST/(n-1)} = 1 - (1-R^{2})\frac{n-1}{n-p-1}.$$

8. What is the key conceptual difference between Adjusted $R^{2}$ and ordinary $R^{2}$?

   Adjusted $R^{2}$ penalizes the addition of predictors by using degrees of freedom; it increases only if a new predictor improves the model more than expected by chance, whereas $R^{2}$ always increases (or stays equal) when predictors are added.

9. Can Adjusted $R^{2}$ be negative? When?

   Yes. Adjusted $R^{2}$ can be negative when the model fits the data very poorly, specifically when $(1-R^{2})\frac{n-1}{n-p-1} > 1$, i.e. the model is worse than the mean-only model after the complexity penalty.

10. What is the relationship between $\bar{R}^{2}$ and $R^{2}$ (which is larger)?

    $\bar{R}^{2} \leq R^{2}$ always (with equality only when $p=0$). Because $\frac{n-1}{n-p-1} \geq 1$, the adjusted value is pulled below the unadjusted one as predictors increase.

11. Express Adjusted $R^{2}$ in terms of the estimated error variance $\hat{\sigma}^{2}$ and the sample variance $s_y^{2}$ of $y$.

    $\bar{R}^{2} = 1 - \dfrac{\hat{\sigma}^{2}}{s_y^{2}}$, where $\hat{\sigma}^{2} = \frac{SSE}{n-p-1}$ and $s_y^{2} = \frac{SST}{n-1}$. Thus maximizing $\bar{R}^{2}$ is equivalent to minimizing $\hat{\sigma}^{2}$.

12. When adding a predictor, what determines whether Adjusted $R^{2}$ increases?

    Adjusted $R^{2}$ increases if and only if the $t$-statistic for the added coefficient exceeds $1$ in absolute value (equivalently, the partial $F$-statistic exceeds $1$); otherwise it decreases.

13. The 'unexplained' variance fraction equals $1-R^{2}$. What is its name and formula in terms of sums of squares?

    It is the fraction of variation left unexplained: $1 - R^{2} = \frac{SSE}{SST}$. It is sometimes called the coefficient of nondetermination.

14. State the multiple linear regression model in matrix form, with assumptions on the error vector.

    $\vec{y} = X\vec{\beta} + \vec{\varepsilon}$, where $X$ is $n\times(p+1)$ of full rank, $E[\vec{\varepsilon}] = \vec{0}$, and $\operatorname{Cov}(\vec{\varepsilon}) = \sigma^{2} I$ (homoscedastic, uncorrelated errors).

15. What is the ordinary least squares (OLS) estimator $\hat{\vec{\beta}}$ in matrix form?

    $\hat{\vec{\beta}} = (X^{T}X)^{-1} X^{T} \vec{y}$, obtained by minimizing $\|\vec{y} - X\vec{\beta}\|^{2}$.

16. State the Gauss-Markov Theorem.

    Under the linear model $\vec{y}=X\vec{\beta}+\vec{\varepsilon}$ with $E[\vec{\varepsilon}]=\vec{0}$ and $\operatorname{Cov}(\vec{\varepsilon})=\sigma^{2}I$, the OLS estimator $\hat{\vec{\beta}}$ is the Best Linear Unbiased Estimator (BLUE): among all linear unbiased estimators of $\vec{\beta}$, it has minimum variance.

17. What does 'BLUE' stand for in the Gauss-Markov Theorem?

    Best (minimum variance) Linear (linear function of $\vec{y}$) Unbiased Estimator. 'Best' means smallest variance within the class of linear unbiased estimators.

18. List the assumptions required for the Gauss-Markov Theorem to hold.

    (1) Linearity: $E[\vec{y}]=X\vec{\beta}$; (2) Zero-mean errors: $E[\vec{\varepsilon}]=\vec{0}$; (3) Homoscedasticity: $\operatorname{Var}(\varepsilon_i)=\sigma^{2}$; (4) Uncorrelated errors: $\operatorname{Cov}(\varepsilon_i,\varepsilon_j)=0$ for $i\neq j$; (5) $X$ has full column rank. Normality is NOT required.

19. Does the Gauss-Markov Theorem require the errors to be normally distributed?

    No. Gauss-Markov requires only zero mean, constant variance, and uncorrelated errors. Normality is needed for exact $t$/$F$ inference and for OLS to be the minimum-variance estimator among ALL (not just linear) unbiased estimators.

20. Under Gauss-Markov assumptions, what is the covariance matrix of the OLS estimator $\hat{\vec{\beta}}$?

    $\operatorname{Cov}(\hat{\vec{\beta}}) = \sigma^{2}(X^{T}X)^{-1}$, and $\hat{\vec{\beta}}$ is unbiased: $E[\hat{\vec{\beta}}]=\vec{\beta}$.

21. For any estimable linear combination $\vec{c}^{T}\vec{\beta}$, what does Gauss-Markov guarantee about $\vec{c}^{T}\hat{\vec{\beta}}$?

    $\vec{c}^{T}\hat{\vec{\beta}}$ is the BLUE of $\vec{c}^{T}\vec{\beta}$: it is unbiased and has the minimum variance among all linear unbiased estimators of $\vec{c}^{T}\vec{\beta}$.

22. What happens to the Gauss-Markov result if errors are heteroscedastic or correlated, i.e. $\operatorname{Cov}(\vec{\varepsilon})=\sigma^{2}V$ with $V\neq I$?

    OLS is still unbiased but no longer BLUE. The BLUE is the Generalized Least Squares (GLS) estimator $\hat{\vec{\beta}}_{GLS} = (X^{T}V^{-1}X)^{-1}X^{T}V^{-1}\vec{y}$ (Aitken's theorem).

23. What is an unbiased estimator of the error variance $\sigma^{2}$ in multiple regression?

    $\hat{\sigma}^{2} = \frac{SSE}{n-p-1} = \frac{\vec{e}^{T}\vec{e}}{n-p-1}$, where $\vec{e}=\vec{y}-X\hat{\vec{\beta}}$ is the residual vector and $n-p-1$ is the residual degrees of freedom. This is the Mean Squared Error (MSE).

24. State the Fisher-Cochran Theorem (Cochran's Theorem).

    Let $\vec{Z}\sim N(\vec{0},I_n)$ and suppose $\vec{Z}^{T}\vec{Z}=\sum_{i=1}^{n}Z_i^{2} = \sum_{j=1}^{k} Q_j$ where each $Q_j=\vec{Z}^{T}A_j\vec{Z}$ is a quadratic form with $\operatorname{rank}(A_j)=r_j$. If $\sum_{j=1}^{k} r_j = n$, then the $Q_j$ are independent and each $Q_j \sim \chi^{2}_{r_j}$.

25. In Cochran's Theorem, what condition on the ranks guarantees independence and chi-square distributions of the quadratic forms?

    The ranks must add up to the total dimension: $\sum_{j} r_j = n$. Equivalently, the symmetric matrices $A_j$ must be idempotent and satisfy $\sum_j A_j = I_n$, which then forces $A_iA_j=0$ for $i\neq j$.

See more Regression Analysis flashcards →

Planning Regression Analysis for GATE Statistics

Regression Analysis (GATE Statistics) FAQ