GATE Statistics Multivariate Analysis Flashcards

24 sample cards from the Multivariate Analysis deck

Sampled from the end of the deck, so these are different cards from the ones shown on the syllabus page.

1. For a random sample of size $n$ from $N_{p}(\vec{\mu}, \Sigma)$, give the distribution of the sample mean vector $\bar{\vec{X}}$.

   $$\bar{\vec{X}} \sim N_{p}\!\left(\vec{\mu},\; \frac{1}{n}\Sigma\right).$$

2. For a sample from $N_{p}(\vec{\mu}, \Sigma)$, what is the relationship (independence) between $\bar{\vec{X}}$ and the sample covariance matrix $S$?

   $\bar{\vec{X}}$ and $S$ are independent. Moreover $(n-1)S \sim W_{p}(n-1, \Sigma)$. This generalizes the univariate independence of $\bar{X}$ and $s^{2}$.

3. Give the maximum likelihood estimators of $\vec{\mu}$ and $\Sigma$ for a sample from $N_{p}(\vec{\mu}, \Sigma)$.

   $$\hat{\vec{\mu}} = \bar{\vec{X}}, \qquad \hat{\Sigma} = \frac{1}{n}\sum_{i=1}^{n}(\vec{X}_i - \bar{\vec{X}})(\vec{X}_i - \bar{\vec{X}})^{\top}.$$ Note $\hat{\Sigma}$ uses divisor $n$, so it is biased; dividing by $n-1$ gives the unbiased $S$.

4. Define Hotelling's $T^{2}$ statistic and state its (scaled) distribution.

   $T^{2} = n(\bar{\vec{X}} - \vec{\mu}_0)^{\top}S^{-1}(\bar{\vec{X}} - \vec{\mu}_0)$. Under $H_0$, $\;\frac{n-p}{(n-1)p}\,T^{2} \sim F_{p,\,n-p}$. It is the multivariate analogue of the squared $t$-statistic.

5. What is the necessary and sufficient condition on $\Sigma$ for the multivariate normal density (non-degenerate form) to exist?

   $\Sigma$ must be symmetric positive-definite (hence $|\Sigma| > 0$ and $\Sigma^{-1}$ exists). If $\Sigma$ is only positive semi-definite (singular), the distribution is degenerate and has no density on $\mathbb{R}^{p}$.

6. Property: If $\vec{X} \sim N_{p}(\vec{\mu}, \Sigma)$ and $A, B$ are constant matrices, when are $A\vec{X}$ and $B\vec{X}$ independent?

   $A\vec{X}$ and $B\vec{X}$ are independent if and only if $A\Sigma B^{\top} = \mathbf{0}$.

7. State the additive property: if $\vec{X} \sim N_{p}(\vec{\mu}_1, \Sigma_1)$ and $\vec{Y} \sim N_{p}(\vec{\mu}_2, \Sigma_2)$ are independent, what is the distribution of $\vec{X} + \vec{Y}$?

   $\vec{X} + \vec{Y} \sim N_{p}(\vec{\mu}_1 + \vec{\mu}_2,\; \Sigma_1 + \Sigma_2)$.

8. What does the principal idea of Principal Component Analysis (PCA) accomplish for a multivariate distribution with covariance $\Sigma$?

   PCA finds orthogonal linear combinations $Y_k = \vec{e}_k^{\top}\vec{X}$ (the principal components) where $\vec{e}_k$ are eigenvectors of $\Sigma$. $\operatorname{Var}(Y_k) = \lambda_k$, the $k$-th eigenvalue, with components ordered $\lambda_1 \geq \lambda_2 \geq \dots \geq \lambda_p$ and mutually uncorrelated.

9. In PCA, what fraction of total variance is explained by the first $k$ principal components?

   $$\frac{\lambda_1 + \lambda_2 + \dots + \lambda_k}{\lambda_1 + \lambda_2 + \dots + \lambda_p} = \frac{\sum_{i=1}^{k}\lambda_i}{\operatorname{tr}(\Sigma)},$$ since total variance equals $\operatorname{tr}(\Sigma) = \sum_i \lambda_i$.

10. For PCA, why is the first principal component the direction of maximum variance?

    It is the unit vector $\vec{e}_1$ maximizing $\vec{a}^{\top}\Sigma\,\vec{a}$ subject to $\vec{a}^{\top}\vec{a} = 1$; the maximizer is the eigenvector of $\Sigma$ with the largest eigenvalue $\lambda_1$, and the maximum value is $\lambda_1$.

11. How do you obtain the correlation matrix $P$ from the covariance matrix $\Sigma$?

    $P = D^{-1/2}\Sigma D^{-1/2}$, where $D = \operatorname{diag}(\sigma_{11}, \dots, \sigma_{pp})$. Equivalently, $\rho_{ij} = \dfrac{\sigma_{ij}}{\sqrt{\sigma_{ii}\sigma_{jj}}}$.

12. Property: If $\vec{X} \sim N_{p}(\vec{0}, I)$ and $C$ is a matrix with $CC^{\top} = \Sigma$, what is the distribution of $\vec{\mu} + C\vec{X}$?

    $\vec{\mu} + C\vec{X} \sim N_{p}(\vec{\mu}, \Sigma)$. This is the standard way to simulate a multivariate normal using a Cholesky factor $C$ of $\Sigma$.

13. Compare the marginal and conditional distributions of a component in a multivariate normal: how do their variances relate?

    The conditional variance is never larger than the marginal variance: $\operatorname{Var}(X_1 \mid \vec{X}_2) \leq \operatorname{Var}(X_1)$, since $\Sigma_{11\cdot 2} = \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$ and the subtracted term is positive semi-definite. Conditioning reduces (or keeps equal) uncertainty.

14. State the relationship $\operatorname{Var}(X) = E[\operatorname{Var}(X\mid Y)] + \operatorname{Var}(E[X\mid Y])$ for the bivariate normal and verify with $\rho$.

    By the law of total variance: $\sigma_X^{2} = \sigma_X^{2}(1-\rho^{2}) + \operatorname{Var}\!\left(\mu_X + \rho\frac{\sigma_X}{\sigma_Y}(Y-\mu_Y)\right) = \sigma_X^{2}(1-\rho^{2}) + \rho^{2}\sigma_X^{2} = \sigma_X^{2}.$ Consistent.

15. What is the joint pdf of the standard bivariate normal $(X,Y)$ with zero means, unit variances and correlation $\rho$?

    $$f(x,y) = \frac{1}{2\pi\sqrt{1-\rho^{2}}}\exp\!\left(-\frac{x^{2} - 2\rho xy + y^{2}}{2(1-\rho^{2})}\right).$$

16. Classification: distinguish a non-degenerate (regular) multivariate normal from a degenerate (singular) one.

    Non-degenerate: $\Sigma$ is positive-definite ($|\Sigma|>0$), a density exists on $\mathbb{R}^{p}$. Degenerate/singular: $\Sigma$ is positive semi-definite but singular ($|\Sigma|=0$); the probability mass lies on a lower-dimensional affine subspace and no density exists w.r.t. Lebesgue measure on $\mathbb{R}^{p}$.

17. Property: For $\vec{X} \sim N_{p}(\vec{\mu}, \Sigma)$, what is $E[(\vec{X}-\vec{\mu})(\vec{X}-\vec{\mu})^{\top}]$ and what are the third central moments?

    $E[(\vec{X}-\vec{\mu})(\vec{X}-\vec{\mu})^{\top}] = \Sigma$. All odd-order central moments are zero (the distribution is symmetric about $\vec{\mu}$), so every third central moment vanishes.

18. How does conditional independence manifest in the inverse covariance (precision) matrix $K = \Sigma^{-1}$ of a multivariate normal?

    $X_i$ and $X_j$ are conditionally independent given all other variables if and only if the $(i,j)$ entry of the precision matrix $K = \Sigma^{-1}$ is zero, i.e. $K_{ij} = 0$. This underlies Gaussian graphical models.

19. Property: If $\vec{X} \sim N_{p}(\vec{\mu}, \Sigma)$, what is the distribution of the orthogonal projection $P\vec{X}$ where $P$ is a symmetric idempotent matrix and $\vec{\mu}=\vec0$, $\Sigma=I$?

    $P\vec{X} \sim N_{p}(\vec{0}, P)$ (degenerate, since $P$ is singular when $P\neq I$), and the quadratic form $\vec{X}^{\top}P\vec{X} \sim \chi^{2}_{r}$ where $r = \operatorname{rank}(P) = \operatorname{tr}(P)$.

20. State the general result on the distribution of a quadratic form $\vec{X}^{\top}A\vec{X}$ when $\vec{X} \sim N_{p}(\vec{0}, I)$.

    $\vec{X}^{\top}A\vec{X} \sim \chi^{2}_{r}$ if and only if $A$ is symmetric idempotent ($A^{2}=A$), with $r = \operatorname{rank}(A)$ degrees of freedom. In general $\vec{X}^{\top}A\vec{X} = \sum_i \lambda_i Z_i^{2}$, a weighted sum of independent $\chi^{2}_{1}$ where $\lambda_i$ are eigenvalues of $A$.

21. What is the canonical correlation analysis (CCA) objective for two random vectors $\vec{X}$ and $\vec{Y}$?

    CCA finds vectors $\vec{a}, \vec{b}$ maximizing the correlation $\operatorname{Corr}(\vec{a}^{\top}\vec{X}, \vec{b}^{\top}\vec{Y})$. The canonical correlations are the square roots of the eigenvalues of $\Sigma_{XX}^{-1}\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}$.

22. Property: Why must the covariance matrix $\Sigma$ of any random vector be positive semi-definite?

    For any constant vector $\vec{a}$, $\;\vec{a}^{\top}\Sigma\,\vec{a} = \operatorname{Var}(\vec{a}^{\top}\vec{X}) \geq 0$. Since a variance cannot be negative, $\Sigma$ is positive semi-definite (and symmetric).

23. For Fisher's linear discriminant between two $N_{p}(\vec{\mu}_1, \Sigma)$ and $N_{p}(\vec{\mu}_2, \Sigma)$ populations, what is the optimal discriminant direction?

    $\vec{w} \propto \Sigma^{-1}(\vec{\mu}_1 - \vec{\mu}_2)$. The linear discriminant function classifies $\vec{x}$ by the sign of $(\vec{\mu}_1-\vec{\mu}_2)^{\top}\Sigma^{-1}\!\left(\vec{x} - \tfrac{1}{2}(\vec{\mu}_1+\vec{\mu}_2)\right)$.

24. Property: For jointly normal $\vec{X}$, summarize why "normality is preserved under marginalization, conditioning, and linear maps."

    Marginals of a multivariate normal are normal; conditionals of a multivariate normal are normal (with linear mean and constant covariance); and any affine transformation $A\vec{X}+\vec{b}$ is normal. These closure properties make the Gaussian family uniquely tractable in multivariate analysis.

What this deck covers

Multivariate Analysis flashcards FAQ