∂ L ∂ Z = y ^ − y \frac{\partial L}{\partial Z}=\hat{y}-y ∂Z∂L=y^−y
其中 y y y为真实值，采用one-hot编码， y ^ \hat{y} y^为softmax输出的预测值
证明： \textbf{证明：} 证明：
根据softmax公式：
y ^ i = e z i ∑ j = 1 n e z j \hat{y}i=\frac{e^{z_i}}{\sum{j=1}^ne^{z_j}} y^i=∑j=1nezjezi
根据CrossEntropy公式：
L = − ∑ i = 1 n y i l o g y ^ i = − ∑ i = 1 n y i l o g e z i ∑ j = 1 n e z j = − y l l o g e z l ∑ j = 1 n e z j − ∑ i = 1 , i ≠ l n y i l o g e z i ∑ j = 1 n e z j \begin{align*} L&=-\sum_{i=1}^ny_ilog\hat{y}i \\ &=-\sum{i=1}^ny_ilog\frac{e^{z_i}}{\sum_{j=1}^ne^{z_j}} \\ &=-y_llog\frac{e^{z_l}}{\sum_{j=1}^ne^{z_j}}-\sum_{i=1,i \neq l}^ny_ilog\frac{e^{z_i}}{\sum_{j=1}^ne^{z_j}} \end{align*} L=−i=1∑nyilogy^i=−i=1∑nyilog∑j=1nezjezi=−yllog∑j=1nezjezl−i=1,i=l∑nyilog∑j=1nezjezi
所以
∂ L ∂ z l = − ∑ i = 1 , i ≠ l n y i ∑ j = 1 n e z j e z i − e z i e z l ( ∑ j = 1 n e z j ) 2 − y l ∑ j = 1 n e z j e z l e z l ∑ j = 1 n e z j − e z l e z l ( ∑ j = 1 n e z j ) 2 = − ∑ i = 1 , i ≠ l n y i ( − e z l ∑ j = 1 n e z j ) − y l ∑ j = 1 n e z j − e z l ∑ j = 1 n e z j = − ∑ i = 1 , i ≠ l n y i ( − e z l ∑ j = 1 n e z j ) − y l + y l e z l ∑ j = 1 n e z j = − y l + ∑ i = 1 n y i e z l ∑ j = 1 n e z j = − y l + e z l ∑ j = 1 n e z j ∑ i = 1 n y i \begin{align*} \frac{\partial L}{\partial z_l} &=-\sum_{i=1,i\neq l}^ny_i\frac{\sum_{j=1}^ne^{z_j}} {e^{z_i}}\frac{-e^{z_i}e^{z_l}}{(\sum_{j=1}^ne^{z_j})^2}-y_l\frac{\sum_{j=1}^ne^{z_j}}{e^{z_l}}\frac{e^{z_l}\sum_{j=1}^ne^{z_j}-e^{z_l}e^{z_l}}{(\sum_{j=1}^ne^{z_j})^2} \\ &=-\sum_{i=1,i\neq l}^ny_i(\frac{-e^{z_l}}{\sum_{j=1}^ne^{z_j}})-y_l\frac{\sum_{j=1}^ne^{z_j}-e^{z_l}}{\sum_{j=1}^ne^{z_j}} \\ &=-\sum_{i=1,i\neq l}^ny_i(\frac{-e^{z_l}}{\sum_{j=1}^ne^{z_j}})-y_l+y_l\frac{e^{z_l}}{\sum_{j=1}^ne^{z_j}} \\ &=-y_l+\sum_{i=1}^ny_i\frac{e^{z_l}}{\sum_{j=1}^ne^{z_j}} \\ &=-y_l+\frac{e^{z_l}}{\sum_{j=1}^ne^{z_j}}\sum_{i=1}^ny_i \end{align*} ∂zl∂L=−i=1,i=l∑nyiezi∑j=1nezj(∑j=1nezj)2−eziezl−ylezl∑j=1nezj(∑j=1nezj)2ezl∑j=1nezj−ezlezl=−i=1,i=l∑nyi(∑j=1nezj−ezl)−yl∑j=1nezj∑j=1nezj−ezl=−i=1,i=l∑nyi(∑j=1nezj−ezl)−yl+yl∑j=1nezjezl=−yl+i=1∑nyi∑j=1nezjezl=−yl+∑j=1nezjezli=1∑nyi
因为 y y y采用one-hot编码，所以
∑ i = 1 n y i = 1 \begin{align*} \sum_{i=1}^ny_i=1 \end{align*} i=1∑nyi=1
所以
∂ L ∂ z l = − y l + e z l ∑ j = 1 n e z j = y ^ l − y l \frac{\partial L}{\partial z_l}=-y_l+\frac{e^{z_l}}{\sum_{j=1}^ne^{z_j}}=\hat{y}_l-y_l ∂zl∂L=−yl+∑j=1nezjezl=y^l−yl
所以
∂ L ∂ Z = y ^ − y \frac{\partial L}{\partial Z}=\hat{y}-y ∂Z∂L=y^−y