经过多次尝试,在kaggle 双T4训练Qwen2.5-0.5B的正确打开方式是:

复制代码

!torchrun --nproc_per_node=2 \
    -m swift.cli.sft \
    --model "./qwen2.5-0.5b-instruct" \
    --dataset /kaggle/working/duan/tools/ai_copilot/sft_dataset.jsonl \
    --max_length 4096 \
    --num_train_epochs 3 \
    --per_device_train_batch_size 4 \
    --learning_rate 5e-5 \
    --output_dir ./output_v2 \
    --logging_steps 5 \
    --save_steps 500 \
    --eval_steps 500 \
    --split_dataset_ratio 0.1 \
    --bf16 true

详细过程

在kaggle上的最佳实践

先安装库

复制代码

!pip install ms-swift[llm] -U -q
!pip install torchao -U -q

下载模型

因为ms-swift自己下载模型太慢,用transformers下载

复制代码

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen2.5-0.5B-Instruct"
save_dir = "./qwen2.5-0.5b-instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.save_pretrained(save_dir)

model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True)
model.save_pretrained(save_dir)

print(f"下载完成，保存在 {save_dir}")

上传训练数据集

我就偷懒了,直接下载段言项目的代码,里面自带数据集

复制代码

!git clone https://gitcode.com/skywalk163/duan/

怎么偷懒呢? 直接让程序帮我们找到数据集的位置

复制代码

import os
for root, dirs, files in os.walk("/kaggle"):
    for f in files:
        if f == "sft_dataset.jsonl":
            print(os.path.join(root, f))

这段代码会自动输出数据集的路径:

复制代码

/kaggle/working/duan/tools/ai_copilot/sft_dataset.jsonl

开始训练

一般我们都是用swift sft 开训,但是在kaggle上双T4卡训练会报错,所以要用torchrun启动:

复制代码

!torchrun --nproc_per_node=2 \
    -m swift.cli.sft \
    --model "./qwen2.5-0.5b-instruct" \
    --dataset /kaggle/working/duan/tools/ai_copilot/sft_dataset.jsonl \
    --max_length 4096 \
    --num_train_epochs 3 \
    --per_device_train_batch_size 4 \
    --learning_rate 5e-5 \
    --output_dir ./output_v2 \
    --logging_steps 5 \
    --save_steps 500 \
    --eval_steps 500 \
    --split_dataset_ratio 0.1 \
    --bf16 true

现在max_length 设为4096, train_batch_size 设为4也能正常训练,不爆显存 .以前用段言自带的训练脚本,max_length 设为2048, train_batch_size 设为1 才能不爆显存!

训练完毕:

复制代码

Train: 100%|██████████████████████████████████| 201/201 [14:58<00:00,  2.82s/it]
{'eval_loss': '0.2401', 'eval_runtime': '8.635', 'eval_samples_per_second': '13.55', 'eval_steps_per_second': '6.833', 'eval_token_acc': '0.9391', 'epoch': '3', 'global_step/max_steps': '201/201', 'elapsed_time': '15m 7s', 'remaining_time': '0s', 'memory(GiB)': '13.83', 'train_speed(s/it)': '4.514'}
Val: 100%|██████████████████████████████████████| 59/59 [00:08<00:00,  7.01it/s]
/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
  return func(*args, **kwargs)
[INFO:swift] Saving model checkpoint to /kaggle/working/output_v2/v2-20260801-011216/checkpoint-201
{'train_runtime': '908.2', 'train_samples_per_second': '3.495', 'train_steps_per_second': '0.221', 'train_loss': '0.3376', 'epoch': '3', 'global_step/max_steps': '201/201', 'elapsed_time': '15m 8s', 'remaining_time': '0s', 'memory(GiB)': '13.83', 'train_speed(s/it)': '4.518'}
Train: 100%|██████████████████████████████████| 201/201 [15:08<00:00,  4.52s/it]
[INFO:swift] last_model_checkpoint: /kaggle/working/output_v2/v2-20260801-011216/checkpoint-201
[INFO:swift] best_model_checkpoint: /kaggle/working/output_v2/v2-20260801-011216/checkpoint-201
[INFO:swift] images_dir: /kaggle/working/output_v2/v2-20260801-011216/images
[INFO:swift] End time of running main: 2026-08-01 01:27:46.525213
[rank0]:[W801 01:27:47.297239747 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

最优模型确认 ：框架（这里使用的是 swift）将 checkpoint-201（即最后一个epoch保存的检查点）标记为最佳模型。

再用6个epoch试试!

有了当前提升准确率的感觉.

训练完成:

复制代码

Val: 100%|██████████████████████████████████████| 59/59 [00:08<00:00,  7.01it/s]
/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
  return func(*args, **kwargs)
[INFO:swift] Saving model checkpoint to /kaggle/working/output_v2/v3-20260801-013448/checkpoint-402
{'train_runtime': '1786', 'train_samples_per_second': '3.554', 'train_steps_per_second': '0.225', 'train_loss': '0.2082', 'epoch': '6', 'global_step/max_steps': '402/402', 'elapsed_time': '29m 46s', 'remaining_time': '0s', 'memory(GiB)': '13.83', 'train_speed(s/it)': '4.443'}
Train: 100%|██████████████████████████████████| 402/402 [29:46<00:00,  4.44s/it]
[INFO:swift] last_model_checkpoint: /kaggle/working/output_v2/v3-20260801-013448/checkpoint-402
[INFO:swift] best_model_checkpoint: /kaggle/working/output_v2/v3-20260801-013448/checkpoint-402
[INFO:swift] images_dir: /kaggle/working/output_v2/v3-20260801-013448/images
[INFO:swift] End time of running main: 2026-08-01 02:04:54.818971
[rank0]:[W801 02:04:55.541631968 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see Redirecting... (function operator())

最后测试下来,500步的效果最好

复制代码

!torchrun --nproc_per_node=2 \
    -m swift.cli.sft \
    --model "./qwen2.5-0.5b-instruct" \
    --dataset /kaggle/working/duan/tools/ai_copilot/sft_dataset.jsonl \
    --max_length 4096 \
    --num_train_epochs 12 \
    --per_device_train_batch_size 4 \
    --learning_rate 5e-5 \
    --output_dir ./output_v2 \
    --logging_steps 5 \
    --save_steps 500 \
    --eval_steps 500 \
    --split_dataset_ratio 0.1 \
    --bf16 true

复制代码

Train: 100%|██████████████████████████████████| 804/804 [47:10<00:00,  2.91s/it]
{'eval_loss': '0.2208', 'eval_runtime': '8.648', 'eval_samples_per_second': '13.53', 'eval_steps_per_second': '6.822', 'eval_token_acc': '0.952', 'epoch': '12', 'global_step/max_steps': '804/804', 'elapsed_time': '47m 19s', 'remaining_time': '0s', 'memory(GiB)': '14.06', 'train_speed(s/it)': '3.531'}
Val: 100%|██████████████████████████████████████| 59/59 [00:08<00:00,  6.98it/s]
/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
  return func(*args, **kwargs)
[INFO:swift] Saving model checkpoint to /kaggle/working/output_v2/v7-20260801-042924/checkpoint-804
{'train_runtime': '2840', 'train_samples_per_second': '4.47', 'train_steps_per_second': '0.283', 'train_loss': '0.1082', 'epoch': '12', 'global_step/max_steps': '804/804', 'elapsed_time': '47m 20s', 'remaining_time': '0s', 'memory(GiB)': '14.06', 'train_speed(s/it)': '3.532'}
Train: 100%|██████████████████████████████████| 804/804 [47:20<00:00,  3.53s/it]
[INFO:swift] last_model_checkpoint: /kaggle/working/output_v2/v7-20260801-042924/checkpoint-804
[INFO:swift] best_model_checkpoint: /kaggle/working/output_v2/v7-20260801-042924/checkpoint-500
[INFO:swift] images_dir: /kaggle/working/output_v2/v7-20260801-042924/images
[INFO:swift] End time of running main: 2026-08-01 05:17:05.974298
[rank0]:[W801 05:17:07.697822352 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

如果爆显存,就把batch_size减小即可.

合并模型

复制代码

!swift merge-lora   --adapters /kaggle/working/output_v2/v7-20260801-042924/checkpoint-500

转为gguf

这步我在kaggle上需要先编译安装llama.cpp,需要较长时间,且经常有报错.所以转gguf我都是在本地让Trae帮我转的.

测试效果

全部完成！v7: 18/18 PASS (100%)，比 v3 的 17/18 有明显提升。

v3 vs v7 对比

#	测试用例	v3	v7	v7 改进点
1	基础函数	OK	OK	补全了参数 `b`
2	多参数默认值	OK	OK	加了 `段落`，参数更完整
3	嵌套条件	OK	OK	加了 `段落` 前缀
4	for-else	OK	OK	`findtarget` 更准确
5	双重循环	OK	OK	`len(mat)` 替代 `N`，变量名正确
6	列表推导	OK	OK	用 `遍历...之...若` 语法！
7	字典推导	OK	OK	接近正确语法
8	类定义	OK	OK	稳定
9	try-except-finally	OK	OK	去掉了 `设 path 为空`
10	类继承+super	OK	OK	多了 `属性 breed`
11	lambda+filter+map	OK	OK	用 `筛选` + `遍历...之...若`
12	match-case	OK	OK	-
13	海象运算符	OK	OK	`(设 n 为 len(data))` 正确！
14	@property	OK	OK	`特性段落 area` 名称正确
15	复合赋值	OK	OK	稳定
16	冒泡排序	OK	OK	`段落 bubble_sort` 完整
17	with语句	FAIL	OK	从失败变通过！
18	装饰器	OK	OK	结构更合理

模型信息

项目	v3	v7
模型名	`duan-translator-v3`	`duan-translator-v7`
训练轮数	402	500
GGUF 大小	948MB	948MB
通过率	17/18 (94.4%)	18/18 (100%)
平均速度	26.6 tok/s	26.1 tok/s

v7 核心改进

列表推导 ：从显式循环升级为 [x 遍历 x 之 20至0：如果 x 取余 2 等于 0] 语法
海象运算符 ：正确输出 (设 n 为 len(data)) 形式
with 语句：从直接失败变为可通过
lambda 高阶函数 ：开始使用 筛选/映射 关键字
更少的冗余 设 xxx 为空 声明

先到这里吧,暂时训练告一段落,该想想这个东西怎么用了.