一元线性回归
fθ(x)=θ0+θ1xf_{\theta}(x)=\theta_{0}+\theta_{1}xfθ(x)=θ0+θ1x
目标函数即损失函数
E(θ)=12∑i=1n(y(i)−fθ(x(i)))2E(\theta)=\cfrac{1}{2}\sum_{i=1}^{n}(y^{(i)}-f_{\theta}(x^{(i)}))^2E(θ)=21i=1∑n(y(i)−fθ(x(i)))2
最优化问题
找到使得E(θ)E(\theta)E(θ)的值最小时的θ\thetaθ，此时函数EEE是关于θ\thetaθ的函数，对EEE求导，当函数EEE的导数在θ\thetaθ等于某一个值的时候大于0，函数EEE是递增的，此时想要获取最小的函数值，应该降低θ\thetaθ，此时使用θ\thetaθ-导数(导数为正)；函数EEE的导数在θ\thetaθ等于某一个值的时候小于0，函数EEE是递减的，此时想要获取最小的函数值，应该增加θ\thetaθ，此时使用θ\thetaθ-导数(导数为负)
梯度下降法
η\etaη是学习率，通过多次迭代才会获取到最终使得损失函数E达到最小值时的θ\thetaθ
θ:=θ−ηddθE(θ)\theta := \theta - \eta \cfrac{d}{d_{\theta}}E(\theta)θ:=θ−ηdθdE(θ)
偏导数
由于θ\thetaθ可能会有多个，当存在多个θ\thetaθ时，需要使用偏导数对每个θ\thetaθ单独求导，然后进行梯度下降，相当于从多个维度同时使得损失函数E趋于最小值
θ0:=θ0−η∂E∂θ0θ1:=θ1−η∂E∂θ1 \theta_0 := \theta_0 - \eta \cfrac{\partial E}{\partial \theta_0} \\ \theta_1 := \theta_1 - \eta \cfrac{\partial E}{\partial \theta_1} θ0:=θ0−η∂θ0∂Eθ1:=θ1−η∂θ1∂E
对损失函数求导函数
损失函数为复合函数
E(θ)=12∑i=1n(y(i)−fθ(x(i)))2E(\theta)=\cfrac{1}{2}\sum_{i=1}^{n}(y^{(i)}-f_{\theta}(x^{(i)}))^2E(θ)=21i=1∑n(y(i)−fθ(x(i)))2
对复合函数求导，做以下假设
u=E(θ)v=fθ(x)∂u∂θ=∂u∂v⋅∂v∂θ \begin{split} u &= E(\theta) \\ v &= f_{\theta}(x) \\ \cfrac{\partial u}{\partial \theta} &= \cfrac{\partial u}{\partial v} \cdot \cfrac{\partial v}{\partial \theta} \end{split} uv∂θ∂u=E(θ)=fθ(x)=∂v∂u⋅∂θ∂v
u对v进行求导
∂u∂v=∂∂v(12∑i=1n(y(i)−v)2)=12∑i=1n(∂∂v(y(i)−v)2)=12∑i=1n(∂∂v(y(i)2−2y(i)v+v2))=12∑i=1n(−2y(i)+2v)=∑i=1n(v−y(i)) \begin{split} \cfrac{\partial u}{\partial v} &= \cfrac{\partial}{\partial v} \begin{pmatrix} \cfrac{1}{2} \sum\limits_{i=1}^{n}(y^{(i)}-v)^2 \end{pmatrix} \\ &=\cfrac{1}{2} \sum\limits_{i=1}^{n} \begin{pmatrix} \cfrac{\partial}{\partial v} (y^{(i)}-v)^2 \end{pmatrix} \\ &=\cfrac{1}{2} \sum\limits_{i=1}^{n} \begin{pmatrix} \cfrac{\partial}{\partial v} (y^{(i)^2}-2y^{(i)}v+v^2) \end{pmatrix} \\ &=\cfrac{1}{2} \sum\limits_{i=1}^{n} \begin{pmatrix} -2y^{(i)}+2v \end{pmatrix} \\ &=\sum\limits_{i=1}^{n} \begin{pmatrix} v-y^{(i)} \end{pmatrix} \\ \end{split} ∂v∂u=∂v∂(21i=1∑n(y(i)−v)2)=21i=1∑n(∂v∂(y(i)−v)2)=21i=1∑n(∂v∂(y(i)2−2y(i)v+v2))=21i=1∑n(−2y(i)+2v)=i=1∑n(v−y(i))
v对θ0\theta_0θ0进行求导
∂v∂θ0=∂∂θ0(θ0+θ1x)=1 \begin{split} \cfrac{\partial v}{\partial \theta_0} &= \cfrac{\partial}{\partial \theta_0} \begin{pmatrix} \theta_0+\theta_1 x \end{pmatrix} \\ & = 1\\ \end{split} ∂θ0∂v=∂θ0∂(θ0+θ1x)=1
u对θ0\theta_0θ0进行求导
∂u∂θ0=∂u∂v⋅∂v∂θ0=∑i=1n(v−y(i))⋅1=∑i=1n(fθ(x(i))−y(i)) \begin{split} \cfrac{\partial u}{\partial \theta_0} &= \cfrac{\partial u}{\partial v} \cdot \cfrac{\partial v}{\partial \theta_0} \\ &=\sum\limits_{i=1}^{n} \begin{pmatrix} v-y^{(i)} \end{pmatrix} \cdot 1\\ &=\sum\limits_{i=1}^{n} \begin{pmatrix} f_{\theta}(x^{(i)})-y^{(i)} \end{pmatrix}\\ \end{split} ∂θ0∂u=∂v∂u⋅∂θ0∂v=i=1∑n(v−y(i))⋅1=i=1∑n(fθ(x(i))−y(i))
v对θ1\theta_1θ1进行求导
∂v∂θ1=∂∂θ1(θ0+θ1x)=x \begin{split} \cfrac{\partial v}{\partial \theta_1} &= \cfrac{\partial}{\partial \theta_1} \begin{pmatrix} \theta_0+\theta_1 x \end{pmatrix} \\ &= x\\ \end{split} ∂θ1∂v=∂θ1∂(θ0+θ1x)=x
u对 \\theta_1 进行求导
∂u∂θ1=∂u∂v⋅∂v∂θ1=∑i=1n(v−y(i))⋅x(i)=∑i=1n(fθ(x(i))−y(i))⋅x(i) \begin{split} \cfrac{\partial u}{\partial \theta_1} &= \cfrac{\partial u}{\partial v} \cdot \cfrac{\partial v}{\partial \theta_1} \\ &=\sum\limits_{i=1}^{n} \begin{pmatrix} v-y^{(i)} \end{pmatrix} \cdot x^{(i)}\\ &=\sum\limits_{i=1}^{n} \begin{pmatrix} f_{\theta}(x^{(i)})-y^{(i)} \end{pmatrix} \cdot x^{(i)} \\ \end{split} ∂θ1∂u=∂v∂u⋅∂θ1∂v=i=1∑n(v−y(i))⋅x(i)=i=1∑n(fθ(x(i))−y(i))⋅x(i)
更新θ0\theta_0θ0和θ1\theta_1θ1
θ0:=θ0−η∑i=1n(fθ(x(i))−y(i))θ1:=θ1−η∑i=1n(fθ(x(i))−y(i))⋅x(i) \begin{split} \theta_0 :&= \theta_0 - \eta \sum\limits_{i=1}^{n} \begin{pmatrix} f_{\theta}(x^{(i)})-y^{(i)} \end{pmatrix} \\ \theta_1 :&= \theta_1 - \eta\sum\limits_{i=1}^{n} \begin{pmatrix} f_{\theta}(x^{(i)})-y^{(i)} \end{pmatrix} \cdot x^{(i)} \end{split} θ0:θ1:=θ0−ηi=1∑n(fθ(x(i))−y(i))=θ1−ηi=1∑n(fθ(x(i))−y(i))⋅x(i)