CSIR NET Mathematical Science UNIT – 4 Syllabus

How do you obtain the marginal distributions from a joint distribution of $(X,Y)$?

Discrete: $p_X(x) = \sum_{y} p(x,y)$ and $p_Y(y) = \sum_{x} p(x,y)$. Continuous: $f_X(x) = \int_{-\infty}^{\infty} f(x,y)\,dy$ and $f_Y(y) = \int_{-\infty}^{\infty} f(x,y)\,dx$.

Define the conditional density of $Y$ given $X=x$.

For $f_X(x) > 0$, $$f_{Y\mid X}(y \mid x) = \frac{f(x,y)}{f_X(x)}.$$ It is a valid density in $y$ for each fixed $x$, integrating to 1.

When are random variables $X$ and $Y$ independent in terms of their joint distribution?

$X$ and $Y$ are independent iff the joint factorizes: $F(x,y) = F_X(x)F_Y(y)$ for all $x,y$, equivalently $f(x,y) = f_X(x)f_Y(y)$ (or $p(x,y)=p_X(x)p_Y(y)$). Then $f_{Y\mid X}(y\mid x) = f_Y(y)$.

Define the expectation of a random variable in the discrete and continuous cases.

Discrete: $E[X] = \sum_{x} x\,p(x)$. Continuous: $E[X] = \int_{-\infty}^{\infty} x\,f(x)\,dx$, provided the sum/integral converges absolutely. More generally $E[g(X)] = \sum_x g(x)p(x)$ or $\int g(x)f(x)\,dx$ (law of the unconscious statistician).

State the linearity property of expectation and how independence affects $E[XY]$.

Linearity (always holds): $E[aX + bY + c] = aE[X] + bE[Y] + c$. If $X$ and $Y$ are independent, then $E[XY] = E[X]\,E[Y]$ (the converse is not generally true).

Define the $r$-th raw moment and the $r$-th central moment of $X$.

The $r$-th raw moment is $\mu'_r = E[X^{r}]$. The $r$-th central moment is $\mu_r = E[(X - \mu)^{r}]$ where $\mu = E[X]$. In particular $\mu_2 = \operatorname{Var}(X)$.

Give the formula for variance in terms of moments and the additivity rule for independent variables.

$\operatorname{Var}(X) = E[X^{2}] - (E[X])^{2}$. If $X_1,\dots,X_n$ are independent (or just pairwise uncorrelated), $\operatorname{Var}\left(\sum_i X_i\right) = \sum_i \operatorname{Var}(X_i)$. In general $\operatorname{Var}(X+Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X,Y)$.

Define covariance and the correlation coefficient between $X$ and $Y$.

$\operatorname{Cov}(X,Y) = E[(X - E X)(Y - E Y)] = E[XY] - E[X]E[Y]$. The correlation coefficient is $$\rho = \frac{\operatorname{Cov}(X,Y)}{\sqrt{\operatorname{Var}(X)\,\operatorname{Var}(Y)}}, \quad -1 \leq \rho \leq 1.$$

Define skewness and kurtosis in terms of central moments.

Skewness $\gamma_1 = \frac{\mu_3}{\mu_2^{3/2}}$ measures asymmetry. Kurtosis $\beta_2 = \frac{\mu_4}{\mu_2^{2}}$ measures tail heaviness/peakedness; excess kurtosis is $\beta_2 - 3$ (zero for the normal distribution).

Define the moment generating function (MGF) and state how moments are recovered from it.

$M_X(t) = E[e^{tX}]$, when it exists in a neighborhood of $0$. Then $E[X^{r}] = M_X^{(r)}(0)$, the $r$-th derivative evaluated at $t=0$. For independent $X,Y$: $M_{X+Y}(t) = M_X(t)M_Y(t)$.

Define the characteristic function of a random variable $X$.

$$\varphi_X(t) = E[e^{itX}] = \int_{-\infty}^{\infty} e^{itx}\,dF_X(x), \quad t \in \mathbb{R}.$$ It always exists (unlike the MGF) since $|e^{itX}| = 1$, and uniquely determines the distribution.

List the key properties of the characteristic function $\varphi_X(t)$.

(1) $\varphi_X(0) = 1$ and $|\varphi_X(t)| \leq 1$. (2) Uniformly continuous on $\mathbb{R}$. (3) $\varphi_X$ uniquely determines the distribution (uniqueness/inversion theorem). (4) For independent $X,Y$: $\varphi_{X+Y}(t) = \varphi_X(t)\varphi_Y(t)$. (5) $\varphi_{aX+b}(t) = e^{itb}\varphi_X(at)$.

How are moments obtained from the characteristic function?

If $E[|X|^{n}] < \infty$, then $\varphi_X$ is $n$ times differentiable and $$E[X^{k}] = \frac{\varphi_X^{(k)}(0)}{i^{k}}, \quad k = 1,\dots,n.$$

State the characteristic function of the standard normal $N(0,1)$ and of $N(\mu,\sigma^{2})$.

For $N(0,1)$: $\varphi(t) = e^{-t^{2}/2}$. For $N(\mu,\sigma^{2})$: $\varphi(t) = \exp\!\left(i\mu t - \tfrac{1}{2}\sigma^{2}t^{2}\right)$.

State Markov's inequality and its key requirement.

For a non-negative random variable $X$ and any $a > 0$, $$P(X \geq a) \leq \frac{E[X]}{a}.$$ It requires $X \geq 0$ and gives an upper bound on tail probability using only the mean.

State Tchebyshev's (Chebyshev's) inequality.

For a random variable $X$ with mean $\mu$ and finite variance $\sigma^{2}$, for any $k > 0$, $$P(|X - \mu| \geq k) \leq \frac{\sigma^{2}}{k^{2}}.$$ Equivalently $P(|X-\mu| \geq k\sigma) \leq \frac{1}{k^{2}}$.

How is Chebyshev's inequality derived from Markov's inequality?

Apply Markov's inequality to the non-negative random variable $(X-\mu)^{2}$: $$P(|X-\mu| \geq k) = P\big((X-\mu)^{2} \geq k^{2}\big) \leq \frac{E[(X-\mu)^{2}]}{k^{2}} = \frac{\sigma^{2}}{k^{2}}.$$

State Jensen's inequality and the role of convexity.

If $g$ is a convex function and $X$ is a random variable with finite mean, then $$g(E[X]) \leq E[g(X)].$$ If $g$ is concave, the inequality reverses: $g(E[X]) \geq E[g(X)]$. Equality holds iff $g$ is linear on the support of $X$ (or $X$ is degenerate).

Give two standard consequences of Jensen's inequality.

Since $x \mapsto x^{2}$ is convex: $E[X^{2}] \geq (E[X])^{2}$, i.e. variance is non-negative. Since $\log$ is concave: $E[\log X] \leq \log E[X]$ (relating geometric and arithmetic means). Also $|E[X]| \leq E|X|$ from convexity of $|\cdot|$.

State the Cauchy-Schwarz inequality for random variables.

For random variables $X,Y$ with finite second moments, $$\big(E[XY]\big)^{2} \leq E[X^{2}]\,E[Y^{2}].$$ This implies the correlation coefficient satisfies $|\rho| \leq 1$.

Define convergence in probability of a sequence $X_n$ to $X$.

$X_n \xrightarrow{P} X$ if for every $\varepsilon > 0$, $$\lim_{n\to\infty} P(|X_n - X| > \varepsilon) = 0.$$

Define almost sure convergence of $X_n$ to $X$.

$X_n \xrightarrow{a.s.} X$ if $$P\left(\lim_{n\to\infty} X_n = X\right) = 1,$$ i.e. $X_n(\omega) \to X(\omega)$ for all $\omega$ outside a set of probability zero.

Define convergence in distribution (weak convergence) of $X_n$ to $X$.

$X_n \xrightarrow{d} X$ if $F_{X_n}(x) \to F_X(x)$ at every continuity point $x$ of $F_X$. Equivalently, $\varphi_{X_n}(t) \to \varphi_X(t)$ for all $t$ (Lévy continuity theorem).

Define convergence in $r$-th mean ($L^r$ convergence) of $X_n$ to $X$.

$X_n \xrightarrow{L^r} X$ if $E[|X_n - X|^{r}] \to 0$ as $n \to \infty$, for $r \geq 1$. The case $r=2$ is convergence in mean square.

State the hierarchy of the modes of convergence (which implies which).

Almost sure convergence $\Rightarrow$ convergence in probability. Convergence in $L^r$ $\Rightarrow$ convergence in probability. Convergence in probability $\Rightarrow$ convergence in distribution. Convergence in distribution is the weakest; none of the reverse implications hold in general (except convergence in distribution to a constant implies convergence in probability).

State the Weak Law of Large Numbers (WLLN) for i.i.d. variables.

If $X_1, X_2, \dots$ are i.i.d. with finite mean $\mu$, then the sample mean $\bar{X}_n = \frac{1}{n}\sum_{i=1}^{n} X_i$ converges to $\mu$ in probability: $$\bar{X}_n \xrightarrow{P} \mu.$$

State the Strong Law of Large Numbers (SLLN) for i.i.d. variables.

If $X_1, X_2, \dots$ are i.i.d. with finite mean $\mu$ (i.e. $E|X_1| < \infty$), then $$\bar{X}_n = \frac{1}{n}\sum_{i=1}^{n} X_i \xrightarrow{a.s.} \mu.$$

Contrast the Weak and Strong Laws of Large Numbers.

WLLN gives convergence in probability ($\bar X_n \xrightarrow{P} \mu$); SLLN gives the stronger almost sure convergence ($\bar X_n \xrightarrow{a.s.} \mu$). SLLN implies WLLN. Kolmogorov's SLLN requires only $E|X_1| < \infty$, while the simplest WLLN proofs use finite variance via Chebyshev.

State the classical (Lindeberg-Lévy) Central Limit Theorem for the i.i.d. case.

If $X_1, X_2, \dots$ are i.i.d. with mean $\mu$ and finite variance $\sigma^{2} > 0$, then $$\frac{\sqrt{n}\,(\bar{X}_n - \mu)}{\sigma} = \frac{\sum_{i=1}^{n} X_i - n\mu}{\sigma\sqrt{n}} \xrightarrow{d} N(0,1).$$

How does the Central Limit Theorem differ from the Law of Large Numbers in what they describe?

The LLN states that $\bar X_n$ converges to the constant $\mu$ (consistency). The CLT describes the fluctuation of $\bar X_n$ around $\mu$ on the scale $\frac{1}{\sqrt{n}}$, asserting that $\sqrt{n}(\bar X_n - \mu)$ is asymptotically normal $N(0,\sigma^{2})$. The CLT gives the rate/shape of convergence.

Define a stochastic process with finite or countable state space and what its state space refers to.

A stochastic process $\{X_t\}$ is a collection of random variables indexed by $t$ (time). Its state space is the set of possible values the $X_t$ take. A finite state space has finitely many states $\{1,2,\dots,N\}$; a countable state space has countably infinitely many states (e.g. $\{0,1,2,\dots\}$), as in a Markov chain on the non-negative integers.