我们通过一个具体的例子来演示多变量线性回归中的梯度下降算法。
示例数据集
假设我们有一个简单的数据集，包含两个特征和一个目标值：
| (x_1) | (x_2) | (y) |
|---|---|---|
| 1 | 2 | 5 |
| 2 | 3 | 8 |
| 3 | 4 | 11 |
| 4 | 5 | 14 |
我们要训练一个线性回归模型，模型的形式为：
f w , b ( x ) = w 1 ⋅ x 1 + w 2 ⋅ x 2 + b f_{w,b}(x) = w_1 \cdot x_1 + w_2 \cdot x_2 + b fw,b(x)=w1⋅x1+w2⋅x2+b
梯度下降步骤
我们从随机初始化的参数 w 1 w_1 w1、 w 2 w_2 w2 和 b b b 开始，然后通过梯度下降算法迭代地更新这些参数。
初始化
假设：
- 初始权重 w 1 = 0 w_1 = 0 w1=0、 w 2 = 0 w_2 = 0 w2=0
- 初始偏置 b = 0 b = 0 b=0
- 学习率 α = 0.01 \alpha = 0.01 α=0.01
- 迭代次数为 2 次（为了简洁）
计算梯度
我们需要计算每个参数的偏导数，并用这些偏导数来更新参数。
第一次迭代
计算偏导数
-
计算预测值和误差 :
预测值 f w , b ( x ( i ) ) = w 1 ⋅ x 1 ( i ) + w 2 ⋅ x 2 ( i ) + b \text{预测值} \quad f_{w,b}(x^{(i)}) = w_1 \cdot x_1^{(i)} + w_2 \cdot x_2^{(i)} + b 预测值fw,b(x(i))=w1⋅x1(i)+w2⋅x2(i)+b对于每个样本，我们计算预测值和误差：
- 对于第一个样本 (1, 2, 5):
f w , b ( x ( 1 ) ) = 0 ⋅ 1 + 0 ⋅ 2 + 0 = 0 误差 = 0 − 5 = − 5 f_{w,b}(x^{(1)}) = 0 \cdot 1 + 0 \cdot 2 + 0 = 0 \\ \text{误差} \quad = 0 - 5 = -5 fw,b(x(1))=0⋅1+0⋅2+0=0误差=0−5=−5 - 对于第二个样本 (2, 3, 8):
f w , b ( x ( 2 ) ) = 0 ⋅ 2 + 0 ⋅ 3 + 0 = 0 误差 = 0 − 8 = − 8 f_{w,b}(x^{(2)}) = 0 \cdot 2 + 0 \cdot 3 + 0 = 0 \\ \text{误差} \quad = 0 - 8 = -8 fw,b(x(2))=0⋅2+0⋅3+0=0误差=0−8=−8 - 对于第三个样本 (3, 4, 11):
f w , b ( x ( 3 ) ) = 0 ⋅ 3 + 0 ⋅ 4 + 0 = 0 误差 = 0 − 11 = − 11 f_{w,b}(x^{(3)}) = 0 \cdot 3 + 0 \cdot 4 + 0 = 0 \\ \text{误差} \quad = 0 - 11 = -11 fw,b(x(3))=0⋅3+0⋅4+0=0误差=0−11=−11 - 对于第四个样本 (4, 5, 14):
f w , b ( x ( 4 ) ) = 0 ⋅ 4 + 0 ⋅ 5 + 0 = 0 误差 = 0 − 14 = − 14 f_{w,b}(x^{(4)}) = 0 \cdot 4 + 0 \cdot 5 + 0 = 0 \\ \text{误差} \quad = 0 - 14 = -14 fw,b(x(4))=0⋅4+0⋅5+0=0误差=0−14=−14
- 对于第一个样本 (1, 2, 5):
-
计算梯度 :
∂ J ∂ w 1 = 1 m ∑ i = 1 m ( f w , b ( x ( i ) ) − y ( i ) ) ⋅ x 1 ( i ) ∂ J ∂ w 2 = 1 m ∑ i = 1 m ( f w , b ( x ( i ) ) − y ( i ) ) ⋅ x 2 ( i ) ∂ J ∂ b = 1 m ∑ i = 1 m ( f w , b ( x ( i ) ) − y ( i ) ) \frac{\partial J}{\partial w_1} = \frac{1}{m} \sum_{i=1}^{m} (f_{w,b}(x^{(i)}) - y^{(i)}) \cdot x_1^{(i)} \\ \frac{\partial J}{\partial w_2} = \frac{1}{m} \sum_{i=1}^{m} (f_{w,b}(x^{(i)}) - y^{(i)}) \cdot x_2^{(i)} \\ \frac{\partial J}{\partial b} = \frac{1}{m} \sum_{i=1}^{m} (f_{w,b}(x^{(i)}) - y^{(i)}) ∂w1∂J=m1i=1∑m(fw,b(x(i))−y(i))⋅x1(i)∂w2∂J=m1i=1∑m(fw,b(x(i))−y(i))⋅x2(i)∂b∂J=m1i=1∑m(fw,b(x(i))−y(i))我们计算每个参数的梯度：
- 对于 w 1 w_1 w1:
∂ J ∂ w 1 = 1 4 [ ( − 5 ) ⋅ 1 + ( − 8 ) ⋅ 2 + ( − 11 ) ⋅ 3 + ( − 14 ) ⋅ 4 ] = 1 4 ( − 5 − 16 − 33 − 56 ) = 1 4 ( − 110 ) = − 27.5 \frac{\partial J}{\partial w_1} = \frac{1}{4} [(-5) \cdot 1 + (-8) \cdot 2 + (-11) \cdot 3 + (-14) \cdot 4] \\ = \frac{1}{4} (-5 - 16 - 33 - 56) \\ = \frac{1}{4} (-110) \\ = -27.5 ∂w1∂J=41[(−5)⋅1+(−8)⋅2+(−11)⋅3+(−14)⋅4]=41(−5−16−33−56)=41(−110)=−27.5 - 对于 w 2 w_2 w2:
∂ J ∂ w 2 = 1 4 [ ( − 5 ) ⋅ 2 + ( − 8 ) ⋅ 3 + ( − 11 ) ⋅ 4 + ( − 14 ) ⋅ 5 ] = 1 4 ( − 10 − 24 − 44 − 70 ) = 1 4 ( − 148 ) = − 37 \frac{\partial J}{\partial w_2} = \frac{1}{4} [(-5) \cdot 2 + (-8) \cdot 3 + (-11) \cdot 4 + (-14) \cdot 5] \\ = \frac{1}{4} (-10 - 24 - 44 - 70) \\ = \frac{1}{4} (-148) \\ = -37 ∂w2∂J=41[(−5)⋅2+(−8)⋅3+(−11)⋅4+(−14)⋅5]=41(−10−24−44−70)=41(−148)=−37 - 对于 b b b:
∂ J ∂ b = 1 4 ( − 5 − 8 − 11 − 14 ) = 1 4 ( − 38 ) = − 9.5 \frac{\partial J}{\partial b} = \frac{1}{4} (-5 - 8 - 11 - 14) \\ = \frac{1}{4} (-38) \\ = -9.5 ∂b∂J=41(−5−8−11−14)=41(−38)=−9.5
- 对于 w 1 w_1 w1:
-
更新参数 :
w 1 = w 1 − α ∂ J ∂ w 1 = 0 − 0.01 ( − 27.5 ) = 0.275 w 2 = w 2 − α ∂ J ∂ w 2 = 0 − 0.01 ( − 37 ) = 0.37 b = b − α ∂ J ∂ b = 0 − 0.01 ( − 9.5 ) = 0.095 w_1 = w_1 - \alpha \frac{\partial J}{\partial w_1} = 0 - 0.01 (-27.5) = 0.275 \\ w_2 = w_2 - \alpha \frac{\partial J}{\partial w_2} = 0 - 0.01 (-37) = 0.37 \\ b = b - \alpha \frac{\partial J}{\partial b} = 0 - 0.01 (-9.5) = 0.095 w1=w1−α∂w1∂J=0−0.01(−27.5)=0.275w2=w2−α∂w2∂J=0−0.01(−37)=0.37b=b−α∂b∂J=0−0.01(−9.5)=0.095
第二次迭代
重复上述步骤，以更新后的参数 w 1 w_1 w1、 w 2 w_2 w2 和 b b b继续计算新的梯度，并更新参数。以下是简略的计算过程：
-
计算预测值和误差:
- 对于第一个样本 (1, 2, 5):
f w , b ( x ( 1 ) ) = 0.275 ⋅ 1 + 0.37 ⋅ 2 + 0.095 = 1.11 误差 = 1.11 − 5 = − 3.89 f_{w,b}(x^{(1)}) = 0.275 \cdot 1 + 0.37 \cdot 2 + 0.095 = 1.11 \\ \text{误差} = 1.11 - 5 = -3.89 fw,b(x(1))=0.275⋅1+0.37⋅2+0.095=1.11误差=1.11−5=−3.89 - 其他样本类似计算。
- 对于第一个样本 (1, 2, 5):
-
计算梯度:
- 对于 w 1 w_1 w1:
∂ J ∂ w 1 ≈ − 21.23 \frac{\partial J}{\partial w_1} \approx -21.23 ∂w1∂J≈−21.23 - 对于 w 2 w_2 w2:
∂ J ∂ w 2 ≈ − 28.74 \frac{\partial J}{\partial w_2} \approx -28.74 ∂w2∂J≈−28.74 - 对于 b b b:
∂ J ∂ b ≈ − 6.83 \frac{\partial J}{\partial b} \approx -6.83 ∂b∂J≈−6.83
- 对于 w 1 w_1 w1:
-
更新参数 :
w 1 = 0.275 − 0.01 ( − 21.23 ) = 0.4873 w 2 = 0.37 − 0.01 ( − 28.74 ) = 0.6574 b = 0.095 − 0.01 ( − 6.83 ) = 0.1633 w_1 = 0.275 - 0.01 (-21.23) = 0.4873 \\ w_2 = 0.37 - 0.01 (-28.74) = 0.6574 \\ b = 0.095 - 0.01 (-6.83) = 0.1633 w1=0.275−0.01(−21.23)=0.4873w2=0.37−0.01(−28.74)=0.6574b=0.095−0.01(−6.83)=0.1633
代码实现
python
def compute_gradient(X, y, w, b):
m, n = X.shape
dj_dw = np.zeros(n)
dj_db = 0.0
for i in range(m):
error = (np.dot(X[i], w) + b) - y[i]
for j in range(n):
dj_dw[j] += error * X[i][j]
dj_db += error
dj_dw /= m
dj_db /= m
return dj_dw, dj_db
def gradient_descent(X, y, w, b, alpha, num_iters):
for i in range(num_iters):
dj_dw, dj_db = compute_gradient(X, y, w, b)
w -= alpha * dj_dw
b -= alpha * dj_db
return w, b总结
通过以上的迭代过程，我们逐步更新参数 w 1 w_1 w1、 w 2 w_2 w2 和 b b b，使得模型的预测值更加接近目标值。实际中，这个过程通常会重复多次，直到参数收敛。