一维/二维高斯分布的负对数似然推导
参考wikipedia Multivariate normal distribution 及 Normal distribution
基础公式
多元高斯分布公式
f ( X ) = f ( x 1 , x 2 , . . . , x k ) = 1 ( 2 π ) k ∣ ∑ ∣ e − 1 2 ( X − μ ) T ∑ − 1 ( X − μ ) f(X)=f(x_1,x_2,...,x_k)=\frac{1}{\sqrt{{(2\pi)}^k|\sum|}}e^{-\frac{1}{2}(X-\mu)^T\sum^{-1}(X-\mu)} f(X)=f(x1,x2,...,xk)=(2π)k∣∑∣1e−21(X−μ)T∑−1(X−μ)
其中， μ = ( μ x 1 , μ x 2 , . . . , μ x k ) \mu=(\mu_{x_1},\mu_{x_2},...,\mu_{x_k}) μ=(μx1,μx2,...,μxk)
那么一维高斯分布公式
f ( x ) = 1 2 π σ 2 e − ( x − μ x ) 2 2 σ 2 f(x)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu_x)^2}{2\sigma^2} } f(x)=2πσ21e−2σ2(x−μx)2
二维高斯分布公式
f ( x , y ) = 1 2 π ∣ ∑ ∣ e − 1 2 ( x − μ x y − μ y ) ∑ − 1 ( x − μ x y − μ y ) f(x,y)=\frac{1}{2\pi\sqrt{|\sum|}}e^{-\frac{1}{2} \begin{pmatrix}x-\mu_x &y-\mu_y\end{pmatrix}\sum^{-1}\begin{pmatrix}x-\mu_x \\ y-\mu_y\end{pmatrix}}\\ f(x,y)=2π∣∑∣1e−21(x−μxy−μy)∑−1(x−μxy−μy)
为简化推导，后文将 d x = x − μ x d_x=x-\mu_x dx=x−μx 记为x
一维高斯分布的负对数似然
N L L ( x , μ ∣ σ 2 ) = − l o g ( f ( x ) ) = − l o g ( 1 2 π σ 2 e − x 2 2 σ 2 ) = − ( − l o g ( 2 π σ 2 ) − x 2 2 σ 2 ) = l o g ( 2 π ) 2 + l o g ( σ ) + x 2 2 σ 2 NLL(x, \mu|\sigma^2)=-log(f(x))=-log(\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{x^2}{2\sigma^2} })\\ = -(-log(\sqrt{2\pi\sigma^2})-\frac{x^2}{2\sigma^2})\\ =\frac{log(2\pi)}{2}+log(\sigma)+\frac{x^2}{2\sigma^2} NLL(x,μ∣σ2)=−log(f(x))=−log(2πσ21e−2σ2x2)=−(−log(2πσ2)−2σ2x2)=2log(2π)+log(σ)+2σ2x2
二维高斯分布的负对数似然
二维高斯分布公式展开如下
f ( x , y ) = 1 2 π ∣ ∑ ∣ e − 1 2 ( x y ) ∑ − 1 ( x y ) = 1 2 π ∣ ∑ ∣ e − 1 2 x 2 σ y 2 − 2 ρ x y σ x σ y + y 2 σ x 2 d e t ( ∣ ∑ ∣ ) = 1 2 π ∣ ∑ ∣ e − 1 2 x 2 σ y 2 − 2 ρ x y σ x σ y + y 2 σ x 2 ( 1 − ρ 2 ) σ x 2 σ y 2 = 1 2 π σ x σ y 1 − ρ 2 e − 1 2 ( 1 − ρ 2 ) ( x 2 σ x 2 − 2 ρ x y σ x σ y + y 2 σ y 2 ) \begin{align}f(x,y)&=\frac{1}{2\pi\sqrt{|\sum|}}e^{-\frac{1}{2} \begin{pmatrix}x &y\end{pmatrix}\sum^{-1}\begin{pmatrix}x \\ y\end{pmatrix}}\\ &=\frac{1}{2\pi\sqrt{|\sum|}}e^{-\frac{1}{2}\frac{x^2\sigma_y^2-2\rho xy\sigma_x\sigma_y+y^2\sigma_x^2}{det(|\sum|)}}\\ &=\frac{1}{2\pi\sqrt{|\sum|}}e^{-\frac{1}{2}\frac{x^2\sigma_y^2-2\rho xy\sigma_x\sigma_y+y^2\sigma_x^2}{(1-\rho^2)\sigma_x^2\sigma_y^2}}\\ &=\frac{1}{2\pi\sigma_x\sigma_y\sqrt{1-\rho^2}}e^{-\frac{1}{2(1-\rho^2)}(\frac{x^2}{\sigma_x^2}-\frac{2\rho xy}{\sigma_x\sigma_y}+\frac{y^2}{\sigma_y^2})} \end{align} f(x,y)=2π∣∑∣1e−21(xy)∑−1(xy)=2π∣∑∣1e−21det(∣∑∣)x2σy2−2ρxyσxσy+y2σx2=2π∣∑∣1e−21(1−ρ2)σx2σy2x2σy2−2ρxyσxσy+y2σx2=2πσxσy1−ρ21e−2(1−ρ2)1(σx2x2−σxσy2ρxy+σy2y2)
其中，
∑ = ( σ x 2 σ x y σ x y σ y 2 ) = ( σ x 2 ρ σ x σ y ρ σ x σ y σ y 2 ) \sum = \begin{pmatrix}{\sigma_x}^2 & \sigma_{xy}\\ \sigma_{xy} &{\sigma_y}^2 \end{pmatrix} = \begin{pmatrix}{\sigma_x}^2 & \rho\sigma_x\sigma_y\\ \rho\sigma_x\sigma_y &{\sigma_y}^2 \end{pmatrix} ∑=(σx2σxyσxyσy2)=(σx2ρσxσyρσxσyσy2)
∑ − 1 = 1 ∣ ∑ ∣ ( σ y 2 − ρ σ x σ y − ρ σ x σ y σ x 2 ) {\sum}^{-1} = \frac{1}{|\sum|}\begin{pmatrix} \sigma_y^2 & -\rho\sigma_x\sigma_y\\ -\rho\sigma_x\sigma_y &\sigma_x^2\end{pmatrix} ∑−1=∣∑∣1(σy2−ρσxσy−ρσxσyσx2)
那么负对数似然为
− l o g ( f ( x , y ) ) = − l o g ( 1 2 π σ x σ y 1 − ρ 2 e − 1 2 ( 1 − ρ 2 ) ( x 2 σ x 2 − 2 ρ x y σ x σ y + y 2 σ y 2 ) ) = − ( − l o g ( 2 π σ x σ y ) − 1 2 l o g ( 1 − ρ 2 ) + ( − 1 2 ( 1 − ρ 2 ) ( x 2 σ x 2 − 2 ρ x y σ x σ y + y 2 σ y 2 ) ) ) = l o g ( 2 π σ x σ y ) + 1 2 l o g ( 1 − ρ 2 ) + 1 2 ( 1 − ρ 2 ) ( x 2 σ x 2 − 2 ρ x y σ x σ y + y 2 σ y 2 ) \begin{align}-log(f(x,y))&=-log(\frac{1}{2\pi\sigma_x\sigma_y\sqrt{1-\rho^2}}e^{-\frac{1}{2(1-\rho^2)}(\frac{x^2}{\sigma_x^2}-\frac{2\rho xy}{\sigma_x\sigma_y}+\frac{y^2}{\sigma_y^2})})\\ &=-(-log(2\pi\sigma_x\sigma_y)-\frac{1}{2}log(1-\rho^2)+(-\frac{1}{2(1-\rho^2)}(\frac{x^2}{\sigma_x^2}-\frac{2\rho xy}{\sigma_x\sigma_y}+\frac{y^2}{\sigma_y^2})))\\ &=log(2\pi\sigma_x\sigma_y)+\frac{1}{2}log(1-\rho^2)+\frac{1}{2(1-\rho^2)}(\frac{x^2}{\sigma_x^2}-\frac{2\rho xy}{\sigma_x\sigma_y}+\frac{y^2}{\sigma_y^2}) \end{align} −log(f(x,y))=−log(2πσxσy1−ρ21e−2(1−ρ2)1(σx2x2−σxσy2ρxy+σy2y2))=−(−log(2πσxσy)−21log(1−ρ2)+(−2(1−ρ2)1(σx2x2−σxσy2ρxy+σy2y2)))=log(2πσxσy)+21log(1−ρ2)+2(1−ρ2)1(σx2x2−σxσy2ρxy+σy2y2)