线性回归分析-HW7

HW.

Prob1 KNNL5.4 KNNL5.12

5.4

We have

$$Y= \begin{bmatrix} 7.8 & 9.0 & 10.2 & 11.0 & 11.7 \end{bmatrix}^T\\ X= \begin{bmatrix} 1 & 1 & 1 & 1 & 1 \\ 8 & 4 & 0 & -4 & -8 \end{bmatrix}^T$$

therefore

$$Y^TY=503.77\\ X^TX= \begin{bmatrix} 5 & 0 \\ 0 & 160 \end{bmatrix}\\ X^TY= \begin{bmatrix} 49.7 \\ -39.2 \end{bmatrix}$$

5.12

$$(X^TX)^{-1}= \begin{bmatrix} \frac{1}{5} & 0 \\ 0 & \frac{1}{160} \end{bmatrix}$$

Prob2 KNNL 5.23

2-a

Estimated regression coefficients

$$b=(X^TX)^{-1}X^TY= \begin{bmatrix} 9.94 \\ -0.245 \end{bmatrix}$$

Vector of residuals

$$e=Y-Xb=\begin{bmatrix} -0.18 & 0.04 & 0.26 & 0.08 & -0.2 \end{bmatrix}^T$$

SSR

$$SSR=\|Xb-\bar{Y}\|^{2}=9.604$$

SSE

$$SSE=\|e||^{2}=0.148$$

Estimated variance-covariance matrix of $b$:

$$\widehat{\mathrm{Cov}}(b)=MSE\cdot(X^TX)^{-1}= \begin{bmatrix} 0.00986667 & 0 \\ 0 &0.00030833 \end{bmatrix}$$

Point estimate of $\mathbb{E}[Y_{h}]$ when $X_{h}=-6$:

$$\hat{Y}_{h}=\begin{bmatrix} 1 & -6 \end{bmatrix}b=11.41$$

Estimated variance of $\hat{Y}_{h}$ when $X_{h}=-6$:

$$\mathbb{V}[\hat{Y}_{h}]=\sigma^{2}\cdot x^T(X^TX)^{-1}X^T\cdot X(X^TX)^{-1}x=\sigma^{2}\cdot x^T(X^TX)^{-1}x\\ \hat{\mathbb{V}}[\hat{Y}_{h}]=MSE\cdot x^T(X^TX)^{-1}x=0.02096$$

2-b

$X^TX$ is diagonal.

2-c

$$H=X(X^TX)^{-1}X^T= \begin{bmatrix} 0.6 & 0.4 & 0.2 & 0.0 & -0.2 \\ 0.4 & 0.3 & 0.2 & 0.1 & 0.0 \\ 0.2 & 0.2 & 0.2 & 0.2 & 0.2 \\ 0.0 & 0.1 & 0.2 & 0.3 & 0.4 \\ -0.2 & 0.0 & 0.2 & 0.4 & 0.6 \end{bmatrix}$$

and $\mathrm{rank}(H)=\mathrm{rank}(X)=2$.

Verify $H$ is idempotent:

2-d

$$\mathrm{Cov}(e)=\sigma^{2}(I-H)$$

therefore

$$s^{2}(e)=MSE\cdot(I-H)= \begin{bmatrix} 0.019733 & -0.019733 & -0.009867 & 0.000000 & 0.009867 \\ -0.019733 & 0.034533 & -0.009867 & -0.004933 & 0.000000 \\ -0.009867 & -0.009867 & 0.039467 & -0.009867 & -0.009867 \\ 0.000000 & -0.004933 & -0.009867 & 0.034533 & -0.019733 \\ 0.009867 & 0.000000 & -0.009867 & -0.019733 & 0.019733 \\ \end{bmatrix}$$

Bonus Question

Let

$$X= \begin{bmatrix} X_{1} & \dots & X_{n} \end{bmatrix}^T,\, a= \begin{bmatrix} a_{1} & \dots & a_{n} \end{bmatrix}^T$$

Then

$$Y=a^TX,\ \mathbb{E}[X]=0,\ \mathrm{Var}(X)= \begin{bmatrix} 1 & p & \dots & p \\ p & 1 & \dots & p \\ \vdots & \vdots & \ddots & \vdots\\ p & p & \dots & 1 \end{bmatrix}$$

So

$$\mathbb{E}[Y]=a^T\mathbb{E}[X]=0\\ \mathbb{V}[Y]=a^T\mathrm{Var}[X]a=a^T((1-p)I+p\boldsymbol{1}\boldsymbol{1}^T)a=(1-p)\sum_{i=1}^{n} a_{i}^{2}+p\left( \sum_{i=1}^{n} a_{i} \right)^{2}$$

Let $S_{1}=\sum a_{i}^{2},S_{2}=\left( \sum a_{i} \right)^{2}$. With Cauchy-Schwarz Inequality we have $nS_{1}\geq S_{2}$. $p$ have to ensure that $\mathbb{V}[Y]\geq 0,\ \forall a\in \mathbb{R}^{n}$.

When $S_{1}\leq S_{2}$, $p$ have to satisfy

$$p\geq \frac{S_{1}}{S_{1}-S_{2}}=\frac{1}{1-S_{2}/S_{1}}\geq \frac{1}{1-n}$$

When $S_{1}>S_{2}$, $p$ have to satisfy

$$p\leq \frac{S_{1}}{S_{1}-S_{2}}$$

which is guaranteed because $p\leq 1$ and $\frac{S_{1}}{S_{1}-S_{2}}> 1$.

So in all, the minimum of $p$ is $\frac{1}{1-n}$.

Think a bit deeper about this conclusion: if we want to impose a shared correlation on $n$ standardized r.v.s, then the imposition cannot be too hard. Take $n=3$ as an example. If we impose a shared correlation of $-1$. Intuitively thinking, $X_{1}$ goes inversely against $X_{2}$, and $X_{2}$ goes inversely against $X_{3}$, then $X_{3}$ should goes consistently with $X_{1}$. So $X_{1}$ and $X_{3}$ cannot be negatively correlated.