作业七
1. 推导逻辑回归代价函数的梯度计算公式
y^=g(z)=hθ(x)=11+e−θTx\hat y=g(z)=h_θ(x)=\frac{1}{1+e^{-θ^Tx}}y^=g(z)=hθ(x)=1+e−θTx1
L(y^,y)=−ylog(y^)−(1−y)log(1−y^)L(\hat y, y)=-ylog(\hat y)-(1-y)log(1-\hat y)L(y^,y)=−ylog(y^)−(1−y)log(1−y^)
J(θ)=1m∑i=1mL(yi^,yi)J(θ)=\frac{1}{m}\sum_{i=1}^{m}L(\hat{y_i},y_i)J(θ)=m1∑i=1mL(yi^,yi)
梯度下降的目标是通过更新参数 θθθ 来最小化代价函数 J(θ)J(θ)J(θ),更新公式为：θj=θj−α⋅∂J(θ)∂θjθ_j=θ_j-\alpha \cdot \frac{\partial J(θ)}{\partial θ_j}θj=θj−α⋅∂θj∂J(θ)
求∂J(θ)∂θj\frac{\partial J(θ)}{\partial θ_j}∂θj∂J(θ)：
∂Li∂θj=∂Li∂yi^⋅∂yi^∂zi⋅∂zi∂θi=(−yiyi^−1−yi1−yi^)⋅(yi^⋅(1−yi^))⋅xi,j=(yi^−yi)⋅xi,j\frac{\partial L_i}{\partial θ_j}=\frac{\partial L_i}{\partial {\hat {y_i}}}\cdot \frac{\partial {\hat {y_i}}}{\partial z_i}\cdot \frac{\partial z_i}{\partial θ_i}=(-\frac{y_i}{\hat{y_i}}-\frac{1-y_i}{1-\hat{y_i}})\cdot (\hat {y_i}\cdot (1-\hat {y_i}))\cdot x_{i,j}=(\hat{y_i}-y_i)\cdot x_{i,j}∂θj∂Li=∂yi^∂Li⋅∂zi∂yi^⋅∂θi∂zi=(−yi^yi−1−yi^1−yi)⋅(yi^⋅(1−yi^))⋅xi,j=(yi^−yi)⋅xi,j
代入得：
θj=θj−α1m∑i=1m(yi^−yi)⋅xi,jθ_j=θ_j-\alpha \frac{1}{m}\sum_{i=1}^{m}(\hat{y_i}-y_i)\cdot x_{i,j}θj=θj−αm1∑i=1m(yi^−yi)⋅xi,j
2. 推导Softmax回归代价函数的梯度计算公式
模型对类别 ccc 的原始得分（logit）为：zc=Wc⋅x=∑j=0nθcjxjz_c=\mathbf{W}c\cdot x=\sum{j=0}^{n}θ_{cj}x_jzc=Wc⋅x=∑j=0nθcjxj
概率分布：yc^=Softmax(z)c=ezc∑c′=1kezc′\hat{y_c}=Softmax(z)c=\frac{e^{z_c}}{\sum{c'=1}^{k}e^zc'}yc^=Softmax(z)c=∑c′=1kezc′ezc
交叉熵损失：L(y^,y)=−∑c=1kyclog(yc^)L(\hat y, y)=-\sum_{c=1}^{k}y_clog(\hat {y_c})L(y^,y)=−∑c=1kyclog(yc^)
总代价函数：J(W)=1m∑i=1mL(y^,y)=−1m∑i=1m∑c=1kyiclog(yic^)J(\mathbf{W})=\frac{1}{m}\sum_{i=1}^{m}L(\hat y, y)=-\frac{1}{m}\sum_{i=1}^{m}\sum_{c=1}^{k}y_{ic}log(\hat {y_{ic}})J(W)=m1∑i=1mL(y^,y)=−m1∑i=1m∑c=1kyiclog(yic^)
梯度推导目标：求 ∂J(W)∂θcj=1m∑i=1m∂Li∂θcj\frac{\partial J(\mathbf{W})}{\partial θ_{cj}}=\frac{1}{m}\sum_{i=1}^{m}\frac{\partial L_i}{\partial θ_{cj}}∂θcj∂J(W)=m1∑i=1m∂θcj∂Li
∂Li∂θcj=−∑c′=1kyic′⋅1yic^⋅∂yic′^∂θcj\frac{\partial L_i}{\partial θ_{cj}}=-\sum_{c'=1}^{k}y_{ic'}\cdot \frac{1}{\hat{y_{ic}}}\cdot \frac{\partial \hat{y_{ic'}}}{\partial θ_{cj}}∂θcj∂Li=−∑c′=1kyic′⋅yic^1⋅∂θcj∂yic′^
∂yic′^∂θcj=\frac{\partial \hat{y_{ic'}}}{\partial θ_{cj}}=∂θcj∂yic′^=
- 当 c′=cc'=cc′=c 时：yic^(1−yic^)⋅xij\hat{y_{ic}}(1-\hat{y_{ic}})\cdot x_{ij}yic^(1−yic^)⋅xij
- 当 c′ !=cc'\ !=cc′ !=c 时：−yic′^yic^⋅xij-\hat{y_{ic'}}\hat{y_{ic}}\cdot x_{ij}−yic′^yic^⋅xij
代入化简得：
∂Li∂θcj=−(yic^−yic)⋅xij\frac{\partial L_i}{\partial θ_{cj}}=-(\hat{y_{ic}}-y_{ic})\cdot x_{ij}∂θcj∂Li=−(yic^−yic)⋅xij
∂J(W)∂θcj=1m∑i=1m(yic^−yic)⋅xij\frac{\partial J(\mathbf{W})}{\partial θ_{cj}}=\frac{1}{m}\sum_{i=1}^{m}(\hat{y_{ic}}-y_{ic})\cdot x_{ij}∂θcj∂J(W)=m1∑i=1m(yic^−yic)⋅xij