《机器学习数学基础》补充资料：连续正态分布随机变量的熵

《机器学习数学基础》第 416 页给出了连续型随机变量的熵的定义，并且在第 417 页以正态分布为例，给出了符合 N ( 0 , σ 2 ) N(0,\sigma^2) N(0,σ2) 的随机变量的熵。

注意：在第 4 次印刷以及之前的版本中，此处有误，具体请阅读勘误表说明

1. 推导（7.6.6）式

假设随机变量服从正态分布 X ∼ N ( μ , σ 2 ) X\sim N(\mu,\sigma^2) X∼N(μ,σ2) （《机器学习数学基础》中是以标准正态分布为例，即 X ∼ N ( 0 , σ 2 ) X\sim N(0,\sigma^2) X∼N(0,σ2) ）。

根据《机器学习数学基础》的（7.6.1）式熵的定义：

H ( X ) = − ∫ f ( x ) log ⁡ f ( x ) d x (7.6.1) H(X)=-\int f(x)\log f(x)\text{d}x\tag{7.6.1} H(X)=−∫f(x)logf(x)dx(7.6.1)

其中， f ( x ) = 1 2 π σ e − ( x − μ ) 2 2 σ 2 f(x)=\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}} f(x)=2π σ1e−2σ2(x−μ)2 ，是概率密度函数。根据均值的定义，（7.6.1）式可以写成：

H ( X ) = − E $log f ( x )$ H(X)=-E $\\log f(x)$ H(X)=−E $logf(x)$

将 f ( x ) f(x) f(x) 代入上式，可得：

H ( X ) = − E $log ( 1 2 π σ e - ( x - μ ) 2 2 σ 2 )$ = − E $log ( 1 2 π σ ) + log ( e - ( x - μ ) 2 2 σ 2 )$ = − E $log ( 1 2 π σ )$ − E $log ( e - ( x - μ ) 2 2 σ 2 )$ = 1 2 log ⁡ ( 2 π σ 2 ) − E $- 1 2 σ 2 ( x - μ ) 2 log e$ = 1 2 log ⁡ ( 2 π σ 2 ) + log ⁡ e 2 σ 2 E $( x - μ ) 2$ = 1 2 log ⁡ ( 2 π σ 2 ) + log ⁡ e 2 σ 2 σ 2 ( ∵ E $( x - μ ) 2$ = σ 2 , 参阅 332 页 ( G 2 ) 式 ) = 1 2 log ⁡ ( 2 π σ 2 ) + 1 2 log ⁡ e = 1 2 log ⁡ ( 2 π e σ 2 ) \begin{split} H(X)&=-E\left $\\log(\\frac{1}{\\sqrt{2\\pi}\\sigma}e\^{-\\frac{(x-\\mu)\^2}{2\\sigma\^2}})\\right$ \\&=-E\left $\\log(\\frac{1}{\\sqrt{2\\pi}\\sigma})+\\log(e\^{-\\frac{(x-\\mu)\^2}{2\\sigma\^2}})\\right$ \\&=-E\left $\\log(\\frac{1}{\\sqrt{2\\pi}\\sigma})\\right$ -E\left $\\log(e\^{-\\frac{(x-\\mu)\^2}{2\\sigma\^2}})\\right$ \\&=\frac{1}{2}\log(2\pi\sigma^2)-E\left $-\\frac{1}{2\\sigma\^2}(x-\\mu)\^2\\log e\\right$ \\&=\frac{1}{2}\log(2\pi\sigma^2)+\frac{\log e}{2\sigma^2}E\left $(x-\\mu)\^2\\right$ \\&=\frac{1}{2}\log(2\pi\sigma^2)+\frac{\log e}{2\sigma^2}\sigma^2\quad(\because E\left $(x-\\mu)\^2\\right$ =\sigma^2,参阅 332 页 (G2)式) \\&=\frac{1}{2}\log(2\pi\sigma^2)+\frac{1}{2}\log e \\&=\frac{1}{2}\log(2\pi e\sigma^2) \end{split} H(X)=−E $log(2π σ1e-2σ2(x-μ)2)$ =−E $log(2π σ1)+log(e-2σ2(x-μ)2)$ =−E $log(2π σ1)$ −E $log(e-2σ2(x-μ)2)$ =21log(2πσ2)−E $-2σ21(x-μ)2loge$ =21log(2πσ2)+2σ2logeE $(x-μ)2$ =21log(2πσ2)+2σ2logeσ2(∵E $(x-μ)2$ =σ2,参阅332页(G2)式)=21log(2πσ2)+21loge=21log(2πeσ2)

从而得到第 417 页（7.6.6）式。

2. 推导多维正态分布的熵

对于服从正态分布的多维随机变量，《机器学习数学基础》中也假设服从标准正态分布，即 X ∼ N ( 0 , Σ ) \pmb{X}\sim N(0,\pmb{\Sigma}) X∼N(0,Σ) 。此处不失一般性，以 X ∼ N ( μ , Σ ) \pmb{X}\sim N(\mu,\pmb{\Sigma}) X∼N(μ,Σ) 为例进行推导。

注意：《机器学习数学基础》第 417 页是以二维随机变量为例，书中明确指出：不妨假设 X = $X 1 X 2$ \pmb{X}=\begin{bmatrix}\pmb{X}_1\\\pmb{X}_2\end{bmatrix} X= $X1X2$ ，因此使用的概率密度函数是第 345 页的（5.5.18）式。

下面的推导，则考虑 n n n 维随机变量，即使用 345 页（5.5.19）式的概率密度函数：

f ( X ) = 1 ( 2 π ) n ∣ Σ ∣ exp ( − 1 2 ( X − μ ) T Σ − 1 ( X − μ ) ) f(\pmb{X})=\frac{1}{\sqrt{(2\pi)^n|\pmb{\Sigma}|}}\text{exp}\left(-\frac{1}{2}(\pmb{X}-\pmb{\mu})^{\text{T}}\pmb{\Sigma}^{-1}(\pmb{X}-\pmb{\mu})\right) f(X)=(2π)n∣Σ∣ 1exp(−21(X−μ)TΣ−1(X−μ))

根据熵的定义（第 416 页（7.6.2）式）得：

H ( X ) = − ∫ f ( X ) log ⁡ ( f ( X ) ) d x = − E $log N ( μ , Σ )$ = − E $log ( ( 2 π ) - n / 2 ∣ Σ ∣ - 1 / 2 exp ( - 1 2 ( X - μ ) T Σ - 1 ( X - μ ) ) )$ = − E $- n 2 log ( 2 π ) - 1 2 log ( ∣ Σ ∣ ) + log exp ( - 1 2 ( X - μ ) T Σ - 1 ( X - μ ) )$ = n 2 log ⁡ ( 2 π ) + 1 2 log ⁡ ( ∣ Σ ∣ ) + log ⁡ e 2 E $( X - μ ) T Σ - 1 ( X - μ )$ \begin{split} H(\pmb{X})&=-\int f(\pmb{X})\log(f(\pmb{X}))\text{d}\pmb{x} \\&=-E\left $\\log N(\\mu,\\pmb{\\Sigma})\\right$ \\&=-E\left $\\log\\left((2\\pi)\^{-n/2}\|\\pmb{\\Sigma}\|\^{-1/2}\\text{exp}\\left(-\\frac{1}{2}(\\pmb{X}-\\pmb{\\mu})\^{\\text{T}}\\pmb{\\Sigma}\^{-1}(\\pmb{X}-\\pmb{\\mu})\\right)\\right)\\right$ \\&=-E\left $-\\frac{n}{2}\\log(2\\pi)-\\frac{1}{2}\\log(\|\\pmb{\\Sigma}\|)+\\log\\text{exp}\\left(-\\frac{1}{2}(\\pmb{X}-\\pmb{\\mu})\^{\\text{T}}\\pmb{\\Sigma}\^{-1}(\\pmb{X}-\\pmb{\\mu})\\right)\\right$ \\&=\frac{n}{2}\log(2\pi)+\frac{1}{2}\log(|\pmb{\Sigma}|)+\frac{\log e}{2}E\left $(\\pmb{X}-\\pmb{\\mu})\^{\\text{T}}\\pmb{\\Sigma}\^{-1}(\\pmb{X}-\\pmb{\\mu})\\right$ \end{split} H(X)=−∫f(X)log(f(X))dx=−E $logN(μ,Σ)$ =−E $log((2π)-n/2∣Σ∣-1/2exp(-21(X-μ)TΣ-1(X-μ)))$ =−E $-2nlog(2π)-21log(∣Σ∣)+logexp(-21(X-μ)TΣ-1(X-μ))$ =2nlog(2π)+21log(∣Σ∣)+2logeE $(X-μ)TΣ-1(X-μ)$

下面单独推导： E $( X - μ ) T Σ - 1 ( X - μ )$ E\left $(\\pmb{X}-\\pmb{\\mu})\^{\\text{T}}\\pmb{\\Sigma}\^{-1}(\\pmb{X}-\\pmb{\\mu})\\right$ E $(X-μ)TΣ-1(X-μ)$ 的值：

E $( X - μ ) T Σ - 1 ( X - μ )$ = E $tr ( ( X - μ ) T Σ - 1 ( X - μ ) )$ = E $tr ( Σ - 1 ( X - μ ) ( X - μ ) T )$ = tr ( Σ − 1 E $( X - μ ) ( X - μ ) T$ ) = tr ( Σ − 1 Σ ) = tr ( I n ) = n \begin{split} E\left $(\\pmb{X}-\\pmb{\\mu})\^{\\text{T}}\\pmb{\\Sigma}\^{-1}(\\pmb{X}-\\pmb{\\mu})\\right$ &=E\left $\\text{tr}\\left((\\pmb{X}-\\pmb{\\mu})\^{\\text{T}}\\pmb{\\Sigma}\^{-1}(\\pmb{X}-\\pmb{\\mu})\\right)\\right$ \\&=E\left $\\text{tr}\\left(\\pmb{\\Sigma}\^{-1}(\\pmb{X}-\\pmb{\\mu})(\\pmb{X}-\\pmb{\\mu})\^{\\text{T}}\\right)\\right$ \\&=\text{tr}\left(\pmb{\Sigma^{-1}}E\left $(\\pmb{X}-\\pmb{\\mu})(\\pmb{X}-\\pmb{\\mu})\^{\\text{T}}\\right$ \right) \\&=\text{tr}(\pmb{\Sigma}^{-1}\pmb{\Sigma}) \\&=\text{tr}(\pmb{I}_n) \\&=n \end{split} E $(X-μ)TΣ-1(X-μ)$ =E $tr((X-μ)TΣ-1(X-μ))$ =E $tr(Σ-1(X-μ)(X-μ)T)$ =tr(Σ−1E $(X-μ)(X-μ)T$ )=tr(Σ−1Σ)=tr(In)=n

所以：

H ( X ) = n 2 log ⁡ ( 2 π ) + 1 2 log ⁡ ( ∣ Σ ∣ ) + log ⁡ e 2 E $( X - μ ) T Σ - 1 ( X - μ )$ = n 2 log ⁡ ( 2 π ) + 1 2 log ⁡ ( ∣ Σ ∣ ) + log ⁡ e 2 n = n 2 ( log ⁡ ( 2 π ) + log ⁡ e ) + 1 2 log ⁡ ( ∣ Σ ∣ ) = n 2 log ⁡ ( 2 π e ) + 1 2 log ⁡ ( ∣ Σ ∣ ) \begin{split} H(\pmb{X})&=\frac{n}{2}\log(2\pi)+\frac{1}{2}\log(|\pmb{\Sigma}|)+\frac{\log e}{2}E\left $(\\pmb{X}-\\pmb{\\mu})\^{\\text{T}}\\pmb{\\Sigma}\^{-1}(\\pmb{X}-\\pmb{\\mu})\\right$ \\&=\frac{n}{2}\log(2\pi)+\frac{1}{2}\log(|\pmb{\Sigma}|)+\frac{\log e}{2}n \\&=\frac{n}{2}\left(\log(2\pi)+\log e\right)+\frac{1}{2}\log(|\pmb{\Sigma}|) \\&=\frac{n}{2}\log(2\pi e)+\frac{1}{2}\log(|\pmb{\Sigma}|) \end{split} H(X)=2nlog(2π)+21log(∣Σ∣)+2logeE $(X-μ)TΣ-1(X-μ)$ =2nlog(2π)+21log(∣Σ∣)+2logen=2n(log(2π)+loge)+21log(∣Σ∣)=2nlog(2πe)+21log(∣Σ∣)

当 n = 2 n=2 n=2 时，即得到《机器学习数学基础》第 417 页推导结果：

H ( X ) = log ⁡ ( 2 π e ) + 1 2 log ⁡ ( ∣ Σ ∣ ) = 1 2 log ⁡ ( ( 2 π e ) 2 ∣ Σ ∣ ) H(\pmb{X})=\log(2\pi e)+\frac{1}{2}\log(|\pmb{\Sigma}|)=\frac{1}{2}\log\left((2\pi e)^2|\pmb{\Sigma|}\right) H(X)=log(2πe)+21log(∣Σ∣)=21log((2πe)2∣Σ∣)

参考资料

$1$ . Entropy of the Gaussian $DB/OL$ . https://gregorygundersen.com/blog/2020/09/01/gaussian-entropy/ , 2023.6.4

$2$ . Entropy and Mutual Information $DB/OL$ . https://gtas.unican.es/files/docencia/TICC/apuntes/tema1bwp_0.pdf ,2023.6.4

$3$ . Fan Cheng. CS258: Information Theory $DB/OL$ . http://qiniu.swarma.org/course/document/lec-7-Differential-Entropy-Part1.pdf , 2023.6.4.

$4$ . Keith Conrad. PROBABILITY DISTRIBUTIONS AND MAXIMUM ENTROPY $DB/OL$ . https://kconrad.math.uconn.edu/blurbs/analysis/entropypost.pdf, 2023.6.4.