【白板推导系列笔记】降维-样本均值样本方差矩阵
X
=
(
x
1
x
2
⋯
x
N
)
N
×
p
T
=
(
x
1
T
x
2
T
⋮
x
N
T
)
=
(
x
11
x
12
⋯
x
1
p
x
21
x
22
⋯
x
2
p
⋮
⋮
⋮
x
N
1
x
N
2
⋯
x
N
P
)
N
×
p
x
i
∈
R
p
,
i
=
1
,
2
,
⋯
,
N
记
1
N
=
(
1
1
⋮
1
)
N
×
1
\begin{gathered} X=\begin{pmatrix} x_{1} & x_{2} & \cdots & x_{N} \end{pmatrix}^{T}_{N \times p}=\begin{pmatrix} x_{1}^{T} \\ x_{2}^{T} \\ \vdots \\ x_{N}^{T} \end{pmatrix}=\begin{pmatrix} x_{11} & x_{12} & \cdots & x_{1p} \\ x_{21} & x_{22} & \cdots & x_{2p} \\ \vdots & \vdots & & \vdots \\ x_{N1} & x_{N2} & \cdots & x_{NP} \end{pmatrix}_{N \times p}\\ x_{i}\in \mathbb{R}^{p},i=1,2,\cdots ,N\\ 记1_{N}=\begin{pmatrix}1 \\ 1 \\ \vdots \\ 1\end{pmatrix}_{N \times 1} \end{gathered}
X=(x1x2⋯xN)N×pT=⎝
⎛x1Tx2T⋮xNT⎠
⎞=⎝
⎛x11x21⋮xN1x12x22⋮xN2⋯⋯⋯x1px2p⋮xNP⎠
⎞N×pxi∈Rp,i=1,2,⋯,N记1N=⎝
⎛11⋮1⎠
⎞N×1
对于样本均值
x
ˉ
=
1
N
∑
i
=
1
N
x
i
=
1
N
(
x
1
x
2
⋯
x
N
)
(
1
1
⋮
1
)
N
×
1
=
1
N
X
T
1
N
\begin{aligned} \bar{x}&=\frac{1}{N}\sum\limits_{i=1}^{N}x_{i}\\ &=\frac{1}{N}\begin{pmatrix} x_{1} & x_{2} & \cdots & x_{N} \end{pmatrix}\begin{pmatrix}1 \\ 1 \\ \vdots \\ 1\end{pmatrix}_{N \times 1}\\ &=\frac{1}{N}X^{T}1_{N} \end{aligned}
xˉ=N1i=1∑Nxi=N1(x1x2⋯xN)⎝
⎛11⋮1⎠
⎞N×1=N1XT1N
对于样本方差
S
=
1
N
∑
i
=
1
N
(
x
i
−
x
ˉ
)
(
x
i
−
x
ˉ
)
T
\begin{aligned} S&=\frac{1}{N}\sum\limits_{i=1}^{N}(x_{i}-\bar{x})(x_{i}-\bar{x})^{T} \end{aligned}
S=N1i=1∑N(xi−xˉ)(xi−xˉ)T
对于
∑
i
=
1
N
(
x
i
−
x
ˉ
)
\sum\limits_{i=1}^{N}(x_{i}-\bar{x})
i=1∑N(xi−xˉ)有
∑
i
=
1
N
(
x
i
−
x
ˉ
)
=
(
x
1
−
x
ˉ
x
2
−
x
ˉ
⋯
x
N
−
x
ˉ
)
=
(
x
1
x
2
⋯
x
N
)
−
(
x
ˉ
x
ˉ
⋯
x
ˉ
)
=
X
T
−
x
ˉ
(
1
1
⋯
1
)
=
X
T
−
x
ˉ
1
N
T
=
X
T
−
1
N
X
T
1
N
1
N
T
=
X
T
(
I
N
−
1
N
1
N
1
N
T
)
\begin{aligned} \sum\limits_{i=1}^{N}(x_{i}-\bar{x})&=\begin{pmatrix} x_{1}-\bar{x} & x_{2}-\bar{x} & \cdots & x_{N}-\bar{x} \end{pmatrix}\\ &=\begin{pmatrix} x_{1} & x_{2} & \cdots & x_{N} \end{pmatrix}-\begin{pmatrix} \bar{x} & \bar{x} & \cdots & \bar{x} \end{pmatrix}\\ &=X^{T}-\bar{x}\begin{pmatrix}1 & 1 & \cdots & 1\end{pmatrix}\\ &=X^{T}-\bar{x}1_{N}^{T}\\ &=X^{T}- \frac{1}{N}X^{T}1_{N}1_{N}^{T}\\ &=X^{T}\left(\mathbb{I}_{N}- \frac{1}{N}1_{N}1_{N}^{T}\right)\\ \end{aligned}
i=1∑N(xi−xˉ)=(x1−xˉx2−xˉ⋯xN−xˉ)=(x1x2⋯xN)−(xˉxˉ⋯xˉ)=XT−xˉ(11⋯1)=XT−xˉ1NT=XT−N1XT1N1NT=XT(IN−N11N1NT)
带回原式
S
=
1
N
(
x
1
−
x
ˉ
x
2
−
x
ˉ
⋯
x
N
−
x
ˉ
)
(
(
x
1
−
x
ˉ
)
T
(
x
2
−
x
ˉ
)
T
⋮
(
x
N
−
x
ˉ
)
T
)
=
1
N
X
T
(
I
N
−
1
N
1
N
1
N
T
)
⋅
(
I
N
−
1
N
1
N
1
N
T
)
T
X
\begin{aligned} S&=\frac{1}{N}\begin{pmatrix} x_{1}-\bar{x} & x_{2}-\bar{x} & \cdots & x_{N}-\bar{x} \end{pmatrix}\begin{pmatrix} (x_{1}-\bar{x})^{T} \\ (x_{2}-\bar{x})^{T} \\ \vdots \\ (x_{N}-\bar{x})^{T} \end{pmatrix}\\ &=\frac{1}{N}X^{T}\left(\mathbb{I}_{N}- \frac{1}{N}1_{N}1_{N}^{T}\right)\cdot (\mathbb{I}_{N}- \frac{1}{N}1_{N}1_{N}^{T})^{T}X\\ \end{aligned}
S=N1(x1−xˉx2−xˉ⋯xN−xˉ)⎝
⎛(x1−xˉ)T(x2−xˉ)T⋮(xN−xˉ)T⎠
⎞=N1XT(IN−N11N1NT)⋅(IN−N11N1NT)TX
记
H
=
I
N
−
1
N
1
N
1
N
T
\begin{aligned} \mathbb{H}=\mathbb{I}_{N}- \frac{1}{N}1_{N}1_{N}^{T}\end{aligned}
H=IN−N11N1NT（
H
\mathbb{H}
H也被称为中心矩阵），上式为
S
=
1
N
X
T
(
I
N
−
1
N
1
N
1
N
T
)
⋅
(
I
N
−
1
N
1
N
1
N
T
)
T
X
=
1
N
X
T
H
⋅
H
X
\begin{aligned} S&=\frac{1}{N}X^{T}\left(\mathbb{I}_{N}- \frac{1}{N}1_{N}1_{N}^{T}\right)\cdot (\mathbb{I}_{N}- \frac{1}{N}1_{N}1_{N}^{T})^{T}X\\ &=\frac{1}{N}X^{T}\mathbb{H}\cdot \mathbb{H}X \end{aligned}
S=N1XT(IN−N11N1NT)⋅(IN−N11N1NT)TX=N1XTH⋅HX
对于
H
T
\mathbb{H}^{T}
HT有
H
T
=
(
I
N
−
1
N
1
N
1
N
T
)
T
=
I
N
−
1
N
1
N
1
N
T
=
H
\begin{aligned} \mathbb{H}^{T}&=(\mathbb{I}_{N}- \frac{1}{N}1_{N}1_{N}^{T})^{T}\\ &=\mathbb{I}_{N}- \frac{1}{N}1_{N}1_{N}^{T}\\ &=\mathbb{H} \end{aligned}
HT=(IN−N11N1NT)T=IN−N11N1NT=H
对于
H
2
\mathbb{H}^{2}
H2有
H
2
=
H
⋅
H
=
(
I
N
−
1
N
1
N
1
N
T
)
(
I
N
−
1
N
1
N
1
N
T
)
=
I
N
−
2
N
1
N
1
N
T
+
1
N
2
1
N
1
N
T
1
N
1
N
T
\begin{aligned} \mathbb{H}^{2}&=\mathbb{H} \cdot \mathbb{H}\\ &=\left(\mathbb{I}_{N}- \frac{1}{N}1_{N}1_{N}^{T}\right)\left(\mathbb{I}_{N}- \frac{1}{N}1_{N}1_{N}^{T}\right)\\ &=\mathbb{I}_{N}- \frac{2}{N}1_{N}1_{N}^{T}+ \frac{1}{N^{2}}1_{N}1_{N}^{T}1_{N}1_{N}^{T} \end{aligned}
H2=H⋅H=(IN−N11N1NT)(IN−N11N1NT)=IN−N21N1NT+N211N1NT1N1NT
对于
1
N
1
N
T
1_{N}1_{N}^{T}
1N1NT
1
N
1
N
T
=
(
1
⋮
1
)
(
1
⋯
1
)
=
(
1
⋯
1
⋮
⋮
1
⋯
1
)
1
N
1
N
T
1
N
1
N
T
=
(
1
⋯
1
⋮
⋮
1
⋯
1
)
(
1
⋯
1
⋮
⋮
1
⋯
1
)
=
(
N
⋯
N
⋮
⋮
N
⋯
N
)
\begin{aligned} 1_{N}1_{N}^{T}&=\begin{pmatrix} 1 \\ \vdots \\ 1 \end{pmatrix}\begin{pmatrix} 1 & \cdots & 1 \end{pmatrix}=\begin{pmatrix} 1 & \cdots & 1 \\ \vdots & & \vdots \\ 1 & \cdots & 1 \end{pmatrix}\\ 1_{N}1_{N}^{T}1_{N}1_{N}^{T}&=\begin{pmatrix} 1 & \cdots & 1 \\ \vdots & & \vdots \\ 1 & \cdots & 1 \end{pmatrix}\begin{pmatrix} 1 & \cdots & 1 \\ \vdots & & \vdots \\ 1 & \cdots & 1 \end{pmatrix}\\ &=\begin{pmatrix} N & \cdots & N \\ \vdots & & \vdots \\ N & \cdots & N \end{pmatrix} \end{aligned}
1N1NT1N1NT1N1NT=⎝
⎛1⋮1⎠
⎞(1⋯1)=⎝
⎛1⋮1⋯⋯1⋮1⎠
⎞=⎝
⎛1⋮1⋯⋯1⋮1⎠
⎞⎝
⎛1⋮1⋯⋯1⋮1⎠
⎞=⎝
⎛N⋮N⋯⋯N⋮N⎠
⎞
带回
H
2
\mathbb{H}^{2}
H2有
H
2
=
I
N
−
2
N
1
N
1
N
T
+
1
N
2
1
N
1
N
T
1
N
1
N
T
=
I
N
−
2
N
(
1
⋯
1
⋮
⋮
1
⋯
1
)
+
1
N
2
(
N
⋯
N
⋮
⋮
N
⋯
N
)
=
I
N
−
2
N
(
1
⋯
1
⋮
⋮
1
⋯
1
)
+
1
N
(
1
⋯
1
⋮
⋮
1
⋯
1
)
=
I
N
−
1
N
(
1
⋯
1
⋮
⋮
1
⋯
1
)
=
I
N
−
1
N
1
N
1
N
T
=
H
\begin{aligned} \mathbb{H}^{2}&=\mathbb{I}_{N}- \frac{2}{N}1_{N}1_{N}^{T}+ \frac{1}{N^{2}}1_{N}1_{N}^{T}1_{N}1_{N}^{T}\\ &=\mathbb{I}_{N}- \frac{2}{N}\begin{pmatrix} 1 & \cdots & 1 \\ \vdots & & \vdots \\ 1 & \cdots & 1 \end{pmatrix}+ \frac{1}{N^{2}}\begin{pmatrix} N & \cdots & N \\ \vdots & & \vdots \\ N & \cdots & N \end{pmatrix}\\ &=\mathbb{I}_{N}- \frac{2}{N}\begin{pmatrix} 1 & \cdots & 1 \\ \vdots & & \vdots \\ 1 & \cdots & 1 \end{pmatrix}+ \frac{1}{N}\begin{pmatrix} 1 & \cdots & 1 \\ \vdots & & \vdots \\ 1 & \cdots & 1 \end{pmatrix}\\ &=\mathbb{I}_{N}- \frac{1}{N}\begin{pmatrix} 1 & \cdots & 1 \\ \vdots & & \vdots \\ 1 & \cdots & 1 \end{pmatrix}\\ &=\mathbb{I}_{N}- \frac{1}{N}1_{N}1_{N}^{T}\\ &=\mathbb{H} \end{aligned}
H2=IN−N21N1NT+N211N1NT1N1NT=IN−N2⎝
⎛1⋮1⋯⋯1⋮1⎠
⎞+N21⎝
⎛N⋮N⋯⋯N⋮N⎠
⎞=IN−N2⎝
⎛1⋮1⋯⋯1⋮1⎠
⎞+N1⎝
⎛1⋮1⋯⋯1⋮1⎠
⎞=IN−N1⎝
⎛1⋮1⋯⋯1⋮1⎠
⎞=IN−N11N1NT=H
因此有
H
n
=
H
\mathbb{H}^{n}=\mathbb{H}
Hn=H，带回
S
S
S
S
=
1
N
X
T
H
⋅
H
X
=
1
N
X
T
H
X
\begin{aligned} S&=\frac{1}{N}X^{T}\mathbb{H}\cdot \mathbb{H}X\\ &=\frac{1}{N}X^{T}\mathbb{H}X \end{aligned}
S=N1XTH⋅HX=N1XTHX
这里中心矩阵
H
\mathbb{H}
H的几何意义是，对于一个数据集
X
X
X，
X
H
X \mathbb{H}
XH可以认为是将数据集平移到坐标轴原点，
H
\mathbb{H}
H就是这个起到平移作用的矩阵