神经网络中 标量求导和向量求导

0. 引出问题

在神经网络反向传播过程中 loss = loss₁,loss₂, loss₃，为什么 ∂loss/∂w

∂loss₁/∂w 
∂loss₂/∂w
∂loss₃/∂w 

∂loss₁/∂w 和 loss 维度一样都是三位向量 ，∂loss₁/∂w, ∂loss₂/∂w, ∂loss₃/∂w 就变成3*3的矩阵

如下所示：

import torch

w = torch.tensor([1.0, 2.0,3.0], requires_grad=True)
loss = w * 3  
print("loss: \n", loss)


loss_m = []

for i, val in enumerate(loss):
    w.grad = None  # 清零
    val.backward(retain_graph=True)
    print(f"∂loss{i+1}/∂w = {w.grad}")
    loss_m.append(w.grad.clone())

print("loss_m: \n", torch.stack(loss_m))

输出结果：

loss: 
 tensor([3., 6., 9.], grad_fn=<MulBackward0>)

∂loss1/∂w = tensor([3., 0., 0.])
∂loss2/∂w = tensor([0., 3., 0.])
∂loss3/∂w = tensor([0., 0., 3.])

loss_m: 
 tensor([[3., 0., 0.],
        [0., 3., 0.],
        [0., 0., 3.]])

loss: tensor(3., 6., 9.) 为向量，对w求导时为矩阵

但是 w.grad 必须 是标量或张量，不能是向量矩阵

1. 标量求导

import torch

w = torch.tensor([1.0, 2.0,3.0], requires_grad=True)
loss = w * 3  
print("loss: \n", loss)


loss_m = []
# 方法1：分别计算
for i, val in enumerate(loss):
    w.grad = None  # 清零
    val.backward(retain_graph=True)
    print(f"∂loss{i+1}/∂w = {w.grad}")
    loss_m.append(w.grad.clone())

print("loss_m: \n", torch.stack(loss_m))

grads = torch.autograd.grad(loss.sum(), w,retain_graph=True)
print("grads: \n", grads)  

grads1 = torch.autograd.grad(loss.mean(), w)[0]
print("grads1: \n", grads1) 

输出；

loss: 
 tensor([3., 6., 9.], grad_fn=<MulBackward0>)
∂loss1/∂w = tensor([3., 0., 0.])
∂loss2/∂w = tensor([0., 3., 0.])
∂loss3/∂w = tensor([0., 0., 3.])
loss_m: 
 tensor([[3., 0., 0.],
        [0., 3., 0.],
        [0., 0., 3.]])
grads: 
 (tensor([3., 3., 3.]),)
grads1: 
 tensor([1., 1., 1.])

同样的例子：

import torch

# 3个样本的真实数据
x = torch.tensor([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]], requires_grad=True)
y_true = torch.tensor([1.0, 2.0, 3.0])

# 线性模型：y = w₁x₁ + w₂x₂
w = torch.tensor([0.5, 0.5], requires_grad=True)
predictions = (x @ w)  # [1.5, 3.5, 5.5]
print("预测值:", predictions)
# 计算每个样本的梯度
individual_grads = []
for i in range(3):
    loss = (predictions[i] - y_true[i])**2
    loss.backward(retain_graph=True)
    individual_grads.append(w.grad.clone())
    w.grad.zero_()

print("样本1梯度:", individual_grads[0]) 
print("样本2梯度:", individual_grads[1])  
print("样本3梯度:", individual_grads[2])  

# 标量梯度：自动综合
total_loss = ((predictions - y_true)**2).mean()
total_loss.backward()


# 验证：标量梯度 = 向量梯度的平均
manual_average = (individual_grads[0] + 
                  individual_grads[1] + 
                  individual_grads[2]) / 3
                  
print("手动平均:", manual_average)  
print("标量结果:", w.grad)  

输出结果：

预测值: tensor([1.5000, 3.5000, 5.5000], grad_fn=<MvBackward0>)
样本1梯度: tensor([1., 2.])
样本2梯度: tensor([ 9., 12.])
样本3梯度: tensor([25., 30.])
手动平均: tensor([11.6667, 14.6667])
标量结果: tensor([11.6667, 14.6667])

训练神经网络是为了最小化整体损失，不是单独优化每个样本

# 实际训练：最小化平均损失
batch_loss = individual_losses.mean()  # 标量
batch_loss.backward()  # 得到平均梯度
optimizer.step()       # 朝平均最优方向更新

2. 什么时候需要向量梯度？

仅用于研究：分析样本敏感性

def compute_sample_gradients(model, x, y):
    """仅用于分析，不用于训练"""
    grads = []
    for xi, yi in zip(x, y):
        model.zero_grad()
        pred = model(xi.unsqueeze(0))
        loss = ((pred - yi) ** 2)
        loss.backward()
        grads.append(model.weight.grad.clone())
    return grads  # 每个样本的单独梯度