Pytorch报错RuntimeError: CUDA error: device-side assert triggered

樱木之2024-03-20 6:05

参考:

https://blog.csdn.net/BetrayFree/article/details/134267306

完整报错如下：

shell 复制代码

Traceback (most recent call last):
  File "/home/yingmuzhi/SpecML2/test.py", line 150, in <module>
    trainer.fit(model, data)
  File "/home/yingmuzhi/SpecML2/core.py", line 486, in fit
    self.fit_epoch()
  File "/home/yingmuzhi/SpecML2/core.py", line 496, in fit_epoch
    loss = self.model.training_step(self.prepare_batch(batch))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yingmuzhi/SpecML2/core.py", line 394, in training_step
    self.plot('loss', l, train=True)
  File "/home/yingmuzhi/SpecML2/core.py", line 384, in plot
    self.board.draw(x, value.to(cpu()).detach().numpy(),
                       ^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

有很多愿意会导致这个CUDA triggered的Error，我的问题是在于网络前半部分都是卷积层，在最后一层使用了全连接层，而在最后一层我的输出并没有和类别相匹配。例如，原先多分类我用的是十个类别，更改成二十二种类别后忘记改输出的类别了，将类别和输出channel相匹配便解决了。