GATE Statistics Estimation Syllabus

Write the general form of a $k$-parameter exponential family density.

$$f(x;\theta) = h(x)\,c(\theta)\,\exp\!\left(\sum_{j=1}^{k} w_j(\theta)\,t_j(x)\right).$$ Here $T(x)=(t_1(x),\dots,t_k(x))$ is the natural sufficient statistic and $w_j(\theta)$ the natural parameters.

For an i.i.d. sample of size $n$ from a $k$-parameter exponential family, what is the natural sufficient statistic?

$T(X) = \left(\sum_{i=1}^{n} t_1(X_i),\ \dots,\ \sum_{i=1}^{n} t_k(X_i)\right)$, the vector of summed natural statistics.

State the completeness theorem for exponential families.

For a full-rank exponential family $$f(x;\eta)=h(x)\,c(\eta)\exp\!\left(\sum_{j=1}^{k}\eta_j t_j(x)\right),$$ if the natural parameter space $\{(\eta_1,\dots,\eta_k)\}$ contains a $k$-dimensional open set (open rectangle), then $T=(t_1,\dots,t_k)$ is complete (and minimal sufficient).

What does 'full rank' mean for an exponential family, and why is it required for completeness?

Full rank means the natural parameter space contains an open $k$-dimensional set (nonempty interior). This curvature/dimensionality guarantees richness of the family, which is the condition ensuring the natural statistic $T$ is complete.

Give an example of a curved exponential family where the natural statistic fails to be complete.

$N(\theta,\theta^2)$ (mean equals standard deviation up to constant) is a curved exponential family: the natural parameter lies on a curve, not an open 2-D set. The minimal sufficient statistic $(\sum X_i,\sum X_i^2)$ is not complete.

Is the natural statistic in a one-parameter full-rank exponential family complete?

Yes. If the natural parameter $\eta$ ranges over an open interval, then $T=\sum_i t(X_i)$ is complete and minimal sufficient for the one-parameter exponential family.

For i.i.d. $\text{Bernoulli}(p)$, identify the complete sufficient statistic and justify via exponential family.

$T=\sum_{i=1}^n X_i$. The Bernoulli pmf is exponential family with natural parameter $\eta=\log\frac{p}{1-p}$, ranging over $\mathbb{R}$ (an open set), so $T$ is complete sufficient for $p$.

For i.i.d. $\text{Poisson}(\lambda)$, give the complete sufficient statistic.

$T=\sum_{i=1}^n X_i$, which is complete sufficient (Poisson is full-rank one-parameter exponential family, $\eta=\log\lambda\in\mathbb{R}$).

For i.i.d. $N(\mu,\sigma^2)$ (both unknown), state the complete sufficient statistic.

$T=\left(\sum_i X_i,\ \sum_i X_i^2\right)$, equivalently $(\bar X, S^2)$. The natural parameter space is a 2-D open set, so $T$ is complete (and minimal) sufficient.

Define an unbiased estimator of $g(\theta)$.

An estimator $W(X)$ is unbiased for $g(\theta)$ if $E_\theta[W(X)] = g(\theta)$ for all $\theta \in \Theta$. Its bias $E_\theta[W]-g(\theta)$ is zero everywhere.

Define the UMVUE (Uniformly Minimum Variance Unbiased Estimator).

$W^*$ is the UMVUE of $g(\theta)$ if it is unbiased and, for every other unbiased estimator $W$ of $g(\theta)$, $$\operatorname{Var}_\theta(W^*) \leq \operatorname{Var}_\theta(W) \quad \text{for all } \theta \in \Theta.$$

State the Rao–Blackwell Theorem.

Let $W$ be an unbiased estimator of $g(\theta)$ and $T$ a sufficient statistic. Define $\phi(T)=E[W\mid T]$. Then $\phi(T)$ is unbiased for $g(\theta)$ and $$\operatorname{Var}_\theta(\phi(T)) \leq \operatorname{Var}_\theta(W)\ \text{for all }\theta,$$ with strict inequality unless $W$ is already a function of $T$.

Why must one condition on a sufficient statistic in the Rao–Blackwell process?

Conditioning on a sufficient statistic ensures $\phi(T)=E[W\mid T]$ does not depend on $\theta$ (so it is a valid statistic/estimator). The sufficiency guarantees the conditional expectation is computable and parameter-free.

State the Lehmann–Scheffé Theorem.

If $T$ is a complete sufficient statistic for $\theta$ and $\phi(T)$ is an unbiased estimator of $g(\theta)$, then $\phi(T)$ is the unique UMVUE of $g(\theta)$ (unique a.s.).

Describe the two-step Lehmann–Scheffé method (Rao–Blackwellization route) to find a UMVUE.

1) Find any unbiased estimator $W$ of $g(\theta)$. 2) Compute $\phi(T)=E[W\mid T]$, where $T$ is complete sufficient. Then $\phi(T)$ is the UMVUE. Completeness guarantees uniqueness.

Describe the direct (functional) method to find a UMVUE.

Find a function $\phi$ of the complete sufficient statistic $T$ such that $E_\theta[\phi(T)] = g(\theta)$ for all $\theta$. By Lehmann–Scheffé, that $\phi(T)$ is the UMVUE. Solving $E[\phi(T)]=g(\theta)$ is often an integral/series equation.

State the characterization of UMVUE via uncorrelatedness with unbiased estimators of zero.

An unbiased estimator $W$ of $g(\theta)$ is the UMVUE if and only if $\operatorname{Cov}_\theta(W,U)=0$ for all $\theta$, for every unbiased estimator $U$ of $0$ (i.e., $E_\theta[U]=0\ \forall\theta$).

Why does completeness guarantee uniqueness of the UMVUE in Lehmann–Scheffé?

If $\phi_1(T)$ and $\phi_2(T)$ are both unbiased for $g(\theta)$, then $E_\theta[\phi_1-\phi_2]=0$ for all $\theta$. Completeness forces $\phi_1=\phi_2$ a.s., so the UMVUE is unique.

Find the UMVUE of $\mu$ for i.i.d. $N(\mu,\sigma^2)$.

$\bar X = \frac{1}{n}\sum_i X_i$. It is unbiased and a function of the complete sufficient statistic, hence the UMVUE of $\mu$.

Find the UMVUE of $\sigma^2$ for i.i.d. $N(\mu,\sigma^2)$ with $\mu$ unknown.

$S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i-\bar X)^2$. It is unbiased for $\sigma^2$ and a function of the complete sufficient statistic, hence the UMVUE.

Find the UMVUE of $\theta$ for i.i.d. $\text{Uniform}(0,\theta)$.

$\dfrac{n+1}{n}X_{(n)}$. Since $E_\theta[X_{(n)}]=\frac{n}{n+1}\theta$ and $X_{(n)}$ is complete sufficient, $\frac{n+1}{n}X_{(n)}$ is the unbiased UMVUE.

Find the UMVUE of $p$ for i.i.d. $\text{Bernoulli}(p)$.

$\bar X = \frac{1}{n}\sum_i X_i$. It is unbiased for $p$ and a function of the complete sufficient statistic $\sum X_i$, hence the UMVUE.

Find the UMVUE of $\lambda$ for i.i.d. $\text{Poisson}(\lambda)$.

$\bar X = \frac{1}{n}\sum_i X_i$, which is unbiased and a function of the complete sufficient statistic $\sum X_i$, hence the UMVUE of $\lambda$.

For i.i.d. $\text{Poisson}(\lambda)$, find the UMVUE of $e^{-\lambda}=P(X=0)$.

$\left(\dfrac{n-1}{n}\right)^{T}$ where $T=\sum_{i=1}^n X_i$. This equals $E[\mathbf{1}(X_1=0)\mid T]$, a function of the complete sufficient statistic, hence the UMVUE of $e^{-\lambda}$.

State the Cramér–Rao Lower Bound (CRLB) for an unbiased estimator of $g(\theta)$.

Under regularity conditions, for any unbiased estimator $W$ of $g(\theta)$, $$\operatorname{Var}_\theta(W) \geq \frac{[g'(\theta)]^{2}}{I_n(\theta)},$$ where $I_n(\theta)$ is the Fisher information of the sample.

Give the formula for Fisher information $I(\theta)$ of a single observation.

$$I(\theta) = E_\theta\!\left[\left(\frac{\partial}{\partial\theta}\log f(X;\theta)\right)^{2}\right] = -E_\theta\!\left[\frac{\partial^{2}}{\partial\theta^{2}}\log f(X;\theta)\right].$$ For an i.i.d. sample of size $n$, $I_n(\theta)=n\,I(\theta)$.

What is the difference between an efficient estimator and the UMVUE?

An efficient estimator attains the CRLB (variance exactly $=$ bound). Every efficient unbiased estimator is the UMVUE, but a UMVUE need not attain the CRLB — the UMVUE can have minimum variance while still exceeding the CRLB.

When does an unbiased estimator attain the CRLB exactly?

Equality in the CRLB holds iff the score factors as $$\frac{\partial}{\partial\theta}\log f(x;\theta) = a(\theta)\,[W(x)-g(\theta)],$$ i.e., the model is exponential-family and $W$ is the natural (linear) estimator. Then $W$ is efficient.

Compare Rao–Blackwell and Lehmann–Scheffé: what does each guarantee?

Rao–Blackwell (sufficiency only): conditioning on a sufficient statistic gives an unbiased estimator with variance no larger — improvement, but not necessarily best. Lehmann–Scheffé (sufficiency + completeness): yields the unique UMVUE. Completeness is the extra ingredient that upgrades 'improved' to 'best and unique'.

Summarize the logical chain linking sufficiency, completeness, and UMVUE.

Sufficiency $\to$ data reduction (Rao–Blackwell improves estimators). Minimal sufficiency $\to$ maximal reduction. Completeness $+$ sufficiency $\to$ unique UMVUE (Lehmann–Scheffé). Full-rank exponential family $\to$ natural statistic is complete sufficient $\to$ UMVUEs are functions of it.