3090微调多模态模型Qwen-VL踩坑

本人使用记录一下训练过程中的心得和bug

1.数据集准备

数据集的标签形式见官方readme,如下:

[
  {
    "id": "identity_0",
    "conversations": [
      {
        "from": "user",
        "value": "你好"
      },
      {
        "from": "assistant",
        "value": "我是Qwen-VL,一个支持视觉输入的大模型。"
      }
    ]
  },
  {
    "id": "identity_1",
    "conversations": [
      {
        "from": "user",
        "value": "Picture 1: <img>https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg</img>\n图中的狗是什么品种？"
      },
      {
        "from": "assistant",
        "value": "图中是一只拉布拉多犬。"
      },
      {
        "from": "user",
        "value": "框出图中的格子衬衫"
      },
      {
        "from": "assistant",
        "value": "<ref>格子衬衫</ref><box>(588,499),(725,789)</box>"
      }
    ]
  },
  { 
    "id": "identity_2",
    "conversations": [
      {
        "from": "user",
        "value": "Picture 1: <img>assets/mm_tutorial/Chongqing.jpeg</img>\nPicture 2: <img>assets/mm_tutorial/Beijing.jpeg</img>\n图中都是哪"
      },
      {
        "from": "assistant",
        "value": "第一张图片是重庆的城市天际线，第二张图片是北京的天际线。"
      }
    ]
  }
]

可以训练纯文本,文本+图,文本+多图(中英文皆可,路径最好绝对路径).将该文件存为label.json.之后进入finetune_qlora_single_gpu.sh将最上面的DATA设置为json文件的路径:

DATA="./label.json"

之后运行finetune_qlora_single_gpu.sh即可开始训练.可以在finetune_qlora_single_gpu.sh中修改epoch参数.3090只能使用qlora微调,多卡训练还没有弄明白.

2.运行报错:"erfinv_vml_cpu" not implemented for 'Half'

见:https://github.com/QwenLM/Qwen-VL/issues/76#issuecomment-1731020993

3.mpi4py安装失败

见:https://blog.csdn.net/weixin_43255962/article/details/103681400

4.auto-gptq安装失败

从源码安装,即:

pip install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/

5.未解决两张卡分别采用finetune_qlora_single_gpu.sh训练两个任务报错:

RuntimeError: The server socket has failed to listen on any local network address. The server socket has failed to bind to [::]:29500 (errno: 98 - Address already in use). The server socket has failed to bind to ?UNKNOWN? (errno: 98 - Address already in use).

问题已经提交github issue: https://github.com/QwenLM/Qwen-VL/issues/187:

其他问题会继续更新,总的来看Qwen-VL比VisualGLM更强.