GATE Statistics Multivariate Analysis Syllabus

Define the population multiple correlation coefficient of $X_1$ with $(X_2, \dots, X_p)$ in a multivariate normal.

$$R_{1\cdot 2\dots p} = \sqrt{\frac{\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}}{\sigma_{11}}},$$ the maximum correlation between $X_1$ and any linear combination of $X_2,\dots,X_p$; it equals the correlation between $X_1$ and its best linear predictor.

Define the partial correlation coefficient $\rho_{12\cdot 3}$ between $X_1$ and $X_2$ given $X_3$.

$$\rho_{12\cdot 3} = \frac{\rho_{12} - \rho_{13}\rho_{23}}{\sqrt{(1-\rho_{13}^{2})(1-\rho_{23}^{2})}}.$$ It measures the correlation between $X_1$ and $X_2$ after removing the linear effect of $X_3$.

What is a key consequence of the conditional-covariance formula regarding partial correlations in a multivariate normal?

Partial correlations are obtained by standardizing the conditional covariance matrix $\Sigma_{11\cdot 2} = \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$. A zero partial correlation corresponds to conditional independence under normality.

Property: What is the distribution of $\vec{X} - \vec{\mu}$ when $\vec{X} \sim N_{p}(\vec{\mu}, \Sigma)$, and is the family closed under translation?

$\vec{X} - \vec{\mu} \sim N_{p}(\vec{0}, \Sigma)$. The multivariate normal family is closed under translation (shifting only changes the mean vector, not $\Sigma$).

State the definition of the Wishart distribution and one role it plays.

If $\vec{X}_1,\dots,\vec{X}_n$ are i.i.d. $N_{p}(\vec{0}, \Sigma)$, then $W = \sum_{i=1}^{n}\vec{X}_i\vec{X}_i^{\top} \sim W_{p}(n, \Sigma)$. It is the multivariate generalization of the chi-square and describes the distribution of the sample covariance (scatter) matrix.

For a random sample of size $n$ from $N_{p}(\vec{\mu}, \Sigma)$, give the distribution of the sample mean vector $\bar{\vec{X}}$.

$$\bar{\vec{X}} \sim N_{p}\!\left(\vec{\mu},\; \frac{1}{n}\Sigma\right).$$

For a sample from $N_{p}(\vec{\mu}, \Sigma)$, what is the relationship (independence) between $\bar{\vec{X}}$ and the sample covariance matrix $S$?

$\bar{\vec{X}}$ and $S$ are independent. Moreover $(n-1)S \sim W_{p}(n-1, \Sigma)$. This generalizes the univariate independence of $\bar{X}$ and $s^{2}$.

Give the maximum likelihood estimators of $\vec{\mu}$ and $\Sigma$ for a sample from $N_{p}(\vec{\mu}, \Sigma)$.

$$\hat{\vec{\mu}} = \bar{\vec{X}}, \qquad \hat{\Sigma} = \frac{1}{n}\sum_{i=1}^{n}(\vec{X}_i - \bar{\vec{X}})(\vec{X}_i - \bar{\vec{X}})^{\top}.$$ Note $\hat{\Sigma}$ uses divisor $n$, so it is biased; dividing by $n-1$ gives the unbiased $S$.

Define Hotelling's $T^{2}$ statistic and state its (scaled) distribution.

$T^{2} = n(\bar{\vec{X}} - \vec{\mu}_0)^{\top}S^{-1}(\bar{\vec{X}} - \vec{\mu}_0)$. Under $H_0$, $\;\frac{n-p}{(n-1)p}\,T^{2} \sim F_{p,\,n-p}$. It is the multivariate analogue of the squared $t$-statistic.

What is the necessary and sufficient condition on $\Sigma$ for the multivariate normal density (non-degenerate form) to exist?

$\Sigma$ must be symmetric positive-definite (hence $|\Sigma| > 0$ and $\Sigma^{-1}$ exists). If $\Sigma$ is only positive semi-definite (singular), the distribution is degenerate and has no density on $\mathbb{R}^{p}$.

Property: If $\vec{X} \sim N_{p}(\vec{\mu}, \Sigma)$ and $A, B$ are constant matrices, when are $A\vec{X}$ and $B\vec{X}$ independent?

$A\vec{X}$ and $B\vec{X}$ are independent if and only if $A\Sigma B^{\top} = \mathbf{0}$.

State the additive property: if $\vec{X} \sim N_{p}(\vec{\mu}_1, \Sigma_1)$ and $\vec{Y} \sim N_{p}(\vec{\mu}_2, \Sigma_2)$ are independent, what is the distribution of $\vec{X} + \vec{Y}$?

$\vec{X} + \vec{Y} \sim N_{p}(\vec{\mu}_1 + \vec{\mu}_2,\; \Sigma_1 + \Sigma_2)$.

What does the principal idea of Principal Component Analysis (PCA) accomplish for a multivariate distribution with covariance $\Sigma$?

PCA finds orthogonal linear combinations $Y_k = \vec{e}_k^{\top}\vec{X}$ (the principal components) where $\vec{e}_k$ are eigenvectors of $\Sigma$. $\operatorname{Var}(Y_k) = \lambda_k$, the $k$-th eigenvalue, with components ordered $\lambda_1 \geq \lambda_2 \geq \dots \geq \lambda_p$ and mutually uncorrelated.

In PCA, what fraction of total variance is explained by the first $k$ principal components?

$$\frac{\lambda_1 + \lambda_2 + \dots + \lambda_k}{\lambda_1 + \lambda_2 + \dots + \lambda_p} = \frac{\sum_{i=1}^{k}\lambda_i}{\operatorname{tr}(\Sigma)},$$ since total variance equals $\operatorname{tr}(\Sigma) = \sum_i \lambda_i$.

For PCA, why is the first principal component the direction of maximum variance?

It is the unit vector $\vec{e}_1$ maximizing $\vec{a}^{\top}\Sigma\,\vec{a}$ subject to $\vec{a}^{\top}\vec{a} = 1$; the maximizer is the eigenvector of $\Sigma$ with the largest eigenvalue $\lambda_1$, and the maximum value is $\lambda_1$.

How do you obtain the correlation matrix $P$ from the covariance matrix $\Sigma$?

$P = D^{-1/2}\Sigma D^{-1/2}$, where $D = \operatorname{diag}(\sigma_{11}, \dots, \sigma_{pp})$. Equivalently, $\rho_{ij} = \dfrac{\sigma_{ij}}{\sqrt{\sigma_{ii}\sigma_{jj}}}$.

Property: If $\vec{X} \sim N_{p}(\vec{0}, I)$ and $C$ is a matrix with $CC^{\top} = \Sigma$, what is the distribution of $\vec{\mu} + C\vec{X}$?

$\vec{\mu} + C\vec{X} \sim N_{p}(\vec{\mu}, \Sigma)$. This is the standard way to simulate a multivariate normal using a Cholesky factor $C$ of $\Sigma$.

Compare the marginal and conditional distributions of a component in a multivariate normal: how do their variances relate?

The conditional variance is never larger than the marginal variance: $\operatorname{Var}(X_1 \mid \vec{X}_2) \leq \operatorname{Var}(X_1)$, since $\Sigma_{11\cdot 2} = \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$ and the subtracted term is positive semi-definite. Conditioning reduces (or keeps equal) uncertainty.

State the relationship $\operatorname{Var}(X) = E[\operatorname{Var}(X\mid Y)] + \operatorname{Var}(E[X\mid Y])$ for the bivariate normal and verify with $\rho$.

By the law of total variance: $\sigma_X^{2} = \sigma_X^{2}(1-\rho^{2}) + \operatorname{Var}\!\left(\mu_X + \rho\frac{\sigma_X}{\sigma_Y}(Y-\mu_Y)\right) = \sigma_X^{2}(1-\rho^{2}) + \rho^{2}\sigma_X^{2} = \sigma_X^{2}.$ Consistent.

What is the joint pdf of the standard bivariate normal $(X,Y)$ with zero means, unit variances and correlation $\rho$?

$$f(x,y) = \frac{1}{2\pi\sqrt{1-\rho^{2}}}\exp\!\left(-\frac{x^{2} - 2\rho xy + y^{2}}{2(1-\rho^{2})}\right).$$

Classification: distinguish a non-degenerate (regular) multivariate normal from a degenerate (singular) one.

Non-degenerate: $\Sigma$ is positive-definite ($|\Sigma|>0$), a density exists on $\mathbb{R}^{p}$. Degenerate/singular: $\Sigma$ is positive semi-definite but singular ($|\Sigma|=0$); the probability mass lies on a lower-dimensional affine subspace and no density exists w.r.t. Lebesgue measure on $\mathbb{R}^{p}$.

Property: For $\vec{X} \sim N_{p}(\vec{\mu}, \Sigma)$, what is $E[(\vec{X}-\vec{\mu})(\vec{X}-\vec{\mu})^{\top}]$ and what are the third central moments?

$E[(\vec{X}-\vec{\mu})(\vec{X}-\vec{\mu})^{\top}] = \Sigma$. All odd-order central moments are zero (the distribution is symmetric about $\vec{\mu}$), so every third central moment vanishes.

How does conditional independence manifest in the inverse covariance (precision) matrix $K = \Sigma^{-1}$ of a multivariate normal?

$X_i$ and $X_j$ are conditionally independent given all other variables if and only if the $(i,j)$ entry of the precision matrix $K = \Sigma^{-1}$ is zero, i.e. $K_{ij} = 0$. This underlies Gaussian graphical models.

Property: If $\vec{X} \sim N_{p}(\vec{\mu}, \Sigma)$, what is the distribution of the orthogonal projection $P\vec{X}$ where $P$ is a symmetric idempotent matrix and $\vec{\mu}=\vec0$, $\Sigma=I$?

$P\vec{X} \sim N_{p}(\vec{0}, P)$ (degenerate, since $P$ is singular when $P\neq I$), and the quadratic form $\vec{X}^{\top}P\vec{X} \sim \chi^{2}_{r}$ where $r = \operatorname{rank}(P) = \operatorname{tr}(P)$.

State the general result on the distribution of a quadratic form $\vec{X}^{\top}A\vec{X}$ when $\vec{X} \sim N_{p}(\vec{0}, I)$.

$\vec{X}^{\top}A\vec{X} \sim \chi^{2}_{r}$ if and only if $A$ is symmetric idempotent ($A^{2}=A$), with $r = \operatorname{rank}(A)$ degrees of freedom. In general $\vec{X}^{\top}A\vec{X} = \sum_i \lambda_i Z_i^{2}$, a weighted sum of independent $\chi^{2}_{1}$ where $\lambda_i$ are eigenvalues of $A$.

What is the canonical correlation analysis (CCA) objective for two random vectors $\vec{X}$ and $\vec{Y}$?

CCA finds vectors $\vec{a}, \vec{b}$ maximizing the correlation $\operatorname{Corr}(\vec{a}^{\top}\vec{X}, \vec{b}^{\top}\vec{Y})$. The canonical correlations are the square roots of the eigenvalues of $\Sigma_{XX}^{-1}\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}$.

Property: Why must the covariance matrix $\Sigma$ of any random vector be positive semi-definite?

For any constant vector $\vec{a}$, $\;\vec{a}^{\top}\Sigma\,\vec{a} = \operatorname{Var}(\vec{a}^{\top}\vec{X}) \geq 0$. Since a variance cannot be negative, $\Sigma$ is positive semi-definite (and symmetric).

For Fisher's linear discriminant between two $N_{p}(\vec{\mu}_1, \Sigma)$ and $N_{p}(\vec{\mu}_2, \Sigma)$ populations, what is the optimal discriminant direction?

$\vec{w} \propto \Sigma^{-1}(\vec{\mu}_1 - \vec{\mu}_2)$. The linear discriminant function classifies $\vec{x}$ by the sign of $(\vec{\mu}_1-\vec{\mu}_2)^{\top}\Sigma^{-1}\!\left(\vec{x} - \tfrac{1}{2}(\vec{\mu}_1+\vec{\mu}_2)\right)$.

Property: For jointly normal $\vec{X}$, summarize why "normality is preserved under marginalization, conditioning, and linear maps."

Marginals of a multivariate normal are normal; conditionals of a multivariate normal are normal (with linear mean and constant covariance); and any affine transformation $A\vec{X}+\vec{b}$ is normal. These closure properties make the Gaussian family uniquely tractable in multivariate analysis.