大模型之基于TRL的ORPO对齐训练实战篇

benben0442026-08-01 18:44

1、环境准备

trl=1.9.2

在32GB * 2 = 64GB显存的环境中训练。

2、数据准备

大模型之基于TRL的DPO对齐训练实战篇-CSDN博客

与上一篇使用一样的train/test数据集。

3、训练

【第一次训练】

代码：

python 复制代码 
import os
os.environ["TRL_EXPERIMENTAL_SILENCE"] = "1"
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig
)
from peft import LoraConfig, get_peft_model
from trl.experimental.orpo import ORPOConfig, ORPOTrainer

# ====================== 配置区 ======================
MODEL_PATH = "/root/autodl-tmp/models/Qwen2.5-7B-Instruct"
TRAIN_DATA = "/root/autodl-tmp/datas/rlhf/med_dpo_answer_train.jsonl"
VAL_DATA = "/root/autodl-tmp/datas/rlhf/med_dpo_answer_test.jsonl"
OUTPUT_DIR = "./orpo-direct-instruct-lora"
LORA_SAVE_PATH = "./orpo-instruct-best-adapter"

# 【非常关键】ORPO eval拼接chosen+rejected，实际≈2*MAX_LENGTH，这里不要超过1024
MAX_LENGTH = 8192

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_PATH,
    quantization_config=bnb_config,
    device_map="balanced",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False})

# ====================== ORPOConfig：打开eval ======================
orpo_config = ORPOConfig(
    output_dir=OUTPUT_DIR,
    num_train_epochs=0.4,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,   # eval batch_size强制1，降低显存尖峰
    gradient_accumulation_steps=4,
    learning_rate=2.0e-6,
    warmup_steps=10,
    bf16=True,
    optim="paged_adamw_8bit",
    lr_scheduler_type="cosine",

    eval_strategy="steps",      # 打开周期性eval
    eval_steps=20,              # 每30 step跑一次验证集，不要设太小
    save_strategy="steps",
    save_steps=20,
    logging_steps=10,
    report_to="none",
    load_best_model_at_end=False,

    beta=0.09,
    max_length=MAX_LENGTH,
)

train_dataset = load_dataset("json", data_files=TRAIN_DATA, split="train")
eval_dataset = load_dataset("json", data_files=VAL_DATA, split="train")

trainer = ORPOTrainer(
    model=model,
    args=orpo_config,
    processing_class=tokenizer,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset
)

# ========== monkey patch：重写evaluate，eval阶段强制开启gradient_checkpointing ==========
raw_evaluate = trainer.evaluate
def patched_evaluate(*args,**kwargs):
    model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant":False})
    model.eval()
    with torch.no_grad():
        metrics = raw_evaluate(*args,**kwargs)
    model.train()
    return metrics

trainer.evaluate = patched_evaluate

# ====================== 启动训练 ======================
trainer.train()

# experimental模块bug，必须peft原生接口保存，不要trainer.save_model()
model.save_pretrained(LORA_SAVE_PATH)
tokenizer.save_pretrained(LORA_SAVE_PATH)

print(f"\n====训练完成====")
print(f"LoRA适配器保存路径：{LORA_SAVE_PATH}")

训练日志：

python 复制代码 
root@autodl-container-4c034b95f8-320f63c6:~/autodl-tmp/codes/sft# python train_orpo_med.py 
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 339/339 [00:02<00:00, 125.55it/s]
trainable params: 40,370,176 || all params: 7,655,986,688 || trainable%: 0.5273
[RANK 0] When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments we have set it for you, but you should do it yourself in the future.
[transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151645}.
{'loss': '1.785', 'grad_norm': '0.2882', 'learning_rate': '1.8e-06', 'rewards/chosen': '-0.152', 'rewards/rejected': '-0.1454', 'rewards/accuracies': '0.225', 'rewards/margins': '-0.006575', 'logps/rejected': '-1.616', 'logps/chosen': '-1.689', 'logits/rejected': '0.2512', 'logits/chosen': '0.6203', 'nll_loss': '1.718', 'log_odds_ratio': '-0.7421', 'log_odds_chosen': '-0.0901', 'epoch': '0.03636'}
{'loss': '1.745', 'grad_norm': '0.3259', 'learning_rate': '1.96e-06', 'rewards/chosen': '-0.1489', 'rewards/rejected': '-0.1386', 'rewards/accuracies': '0.425', 'rewards/margins': '-0.01031', 'logps/rejected': '-1.539', 'logps/chosen': '-1.654', 'logits/rejected': '0.2377', 'logits/chosen': '0.7501', 'nll_loss': '1.675', 'log_odds_ratio': '-0.778', 'log_odds_chosen': '-0.1449', 'epoch': '0.07273'}
{'eval_loss': '1.739', 'eval_runtime': '331.6', 'eval_samples_per_second': '0.902', 'eval_steps_per_second': '0.902', 'eval_rewards/chosen': '-0.1481', 'eval_rewards/rejected': '-0.1422', 'eval_rewards/accuracies': '0.3946', 'eval_rewards/margins': '-0.005906', 'eval_logps/rejected': '-1.58', 'eval_logps/chosen': '-1.646', 'eval_logits/rejected': '0.3267', 'eval_logits/chosen': '0.725', 'eval_nll_loss': '1.672', 'eval_log_odds_ratio': '-0.7416', 'eval_log_odds_chosen': '-0.08309', 'epoch': '0.07273'}
{'loss': '1.747', 'grad_norm': '0.3736', 'learning_rate': '1.827e-06', 'rewards/chosen': '-0.149', 'rewards/rejected': '-0.1408', 'rewards/accuracies': '0.3', 'rewards/margins': '-0.008223', 'logps/rejected': '-1.564', 'logps/chosen': '-1.655', 'logits/rejected': '0.3436', 'logits/chosen': '0.6667', 'nll_loss': '1.679', 'log_odds_ratio': '-0.7574', 'log_odds_chosen': '-0.1148', 'epoch': '0.1091'}  
{'loss': '1.702', 'grad_norm': '0.3392', 'learning_rate': '1.613e-06', 'rewards/chosen': '-0.1452', 'rewards/rejected': '-0.1407', 'rewards/accuracies': '0.475', 'rewards/margins': '-0.00448', 'logps/rejected': '-1.564', 'logps/chosen': '-1.613', 'logits/rejected': '0.2778', 'logits/chosen': '0.7098', 'nll_loss': '1.636', 'log_odds_ratio': '-0.7323', 'log_odds_chosen': '-0.0649', 'epoch': '0.1455'}
{'eval_loss': '1.737', 'eval_runtime': '331.7', 'eval_samples_per_second': '0.901', 'eval_steps_per_second': '0.901', 'eval_rewards/chosen': '-0.1479', 'eval_rewards/rejected': '-0.142', 'eval_rewards/accuracies': '0.388', 'eval_rewards/margins': '-0.005855', 'eval_logps/rejected': '-1.578', 'eval_logps/chosen': '-1.643', 'eval_logits/rejected': '0.3262', 'eval_logits/chosen': '0.7213', 'eval_nll_loss': '1.67', 'eval_log_odds_ratio': '-0.7413', 'eval_log_odds_chosen': '-0.08243', 'epoch': '0.1455'}
{'loss': '1.724', 'grad_norm': '0.3159', 'learning_rate': '1.339e-06', 'rewards/chosen': '-0.1472', 'rewards/rejected': '-0.1426', 'rewards/accuracies': '0.35', 'rewards/margins': '-0.004598', 'logps/rejected': '-1.585', 'logps/chosen': '-1.636', 'logits/rejected': '0.3283', 'logits/chosen': '0.6896', 'nll_loss': '1.659', 'log_odds_ratio': '-0.7307', 'log_odds_chosen': '-0.06555', 'epoch': '0.1818'}
{'loss': '1.755', 'grad_norm': '0.3206', 'learning_rate': '1.031e-06', 'rewards/chosen': '-0.1503', 'rewards/rejected': '-0.1454', 'rewards/accuracies': '0.375', 'rewards/margins': '-0.004956', 'logps/rejected': '-1.615', 'logps/chosen': '-1.67', 'logits/rejected': '0.175', 'logits/chosen': '0.697', 'nll_loss': '1.689', 'log_odds_ratio': '-0.7323', 'log_odds_chosen': '-0.06832', 'epoch': '0.2182'}
{'eval_loss': '1.733', 'eval_runtime': '331.7', 'eval_samples_per_second': '0.902', 'eval_steps_per_second': '0.902', 'eval_rewards/chosen': '-0.1476', 'eval_rewards/rejected': '-0.1418', 'eval_rewards/accuracies': '0.3913', 'eval_rewards/margins': '-0.005802', 'eval_logps/rejected': '-1.575', 'eval_logps/chosen': '-1.64', 'eval_logits/rejected': '0.3257', 'eval_logits/chosen': '0.7161', 'eval_nll_loss': '1.667', 'eval_log_odds_ratio': '-0.7409', 'eval_log_odds_chosen': '-0.08173', 'epoch': '0.2182'}
{'loss': '1.684', 'grad_norm': '0.3391', 'learning_rate': '7.21e-07', 'rewards/chosen': '-0.1437', 'rewards/rejected': '-0.1373', 'rewards/accuracies': '0.325', 'rewards/margins': '-0.006451', 'logps/rejected': '-1.525', 'logps/chosen': '-1.597', 'logits/rejected': '0.3031', 'logits/chosen': '0.7656', 'nll_loss': '1.617', 'log_odds_ratio': '-0.7441', 'log_odds_chosen': '-0.09265', 'epoch': '0.2545'}
{'loss': '1.714', 'grad_norm': '0.3033', 'learning_rate': '4.379e-07', 'rewards/chosen': '-0.1461', 'rewards/rejected': '-0.1372', 'rewards/accuracies': '0.2', 'rewards/margins': '-0.008878', 'logps/rejected': '-1.525', 'logps/chosen': '-1.623', 'logits/rejected': '0.2312', 'logits/chosen': '0.6741', 'nll_loss': '1.645', 'log_odds_ratio': '-0.7627', 'log_odds_chosen': '-0.1272', 'epoch': '0.2909'}
{'eval_loss': '1.731', 'eval_runtime': '332.1', 'eval_samples_per_second': '0.9', 'eval_steps_per_second': '0.9', 'eval_rewards/chosen': '-0.1474', 'eval_rewards/rejected': '-0.1416', 'eval_rewards/accuracies': '0.388', 'eval_rewards/margins': '-0.005768', 'eval_logps/rejected': '-1.573', 'eval_logps/chosen': '-1.638', 'eval_logits/rejected': '0.3231', 'eval_logits/chosen': '0.7121', 'eval_nll_loss': '1.664', 'eval_log_odds_ratio': '-0.7407', 'eval_log_odds_chosen': '-0.0813', 'epoch': '0.2909'}
{'loss': '1.699', 'grad_norm': '0.331', 'learning_rate': '2.098e-07', 'rewards/chosen': '-0.1449', 'rewards/rejected': '-0.1406', 'rewards/accuracies': '0.425', 'rewards/margins': '-0.004294', 'logps/rejected': '-1.562', 'logps/chosen': '-1.609', 'logits/rejected': '0.2502', 'logits/chosen': '0.6972', 'nll_loss': '1.633', 'log_odds_ratio': '-0.7305', 'log_odds_chosen': '-0.06163', 'epoch': '0.3273'}
{'loss': '1.727', 'grad_norm': '0.3122', 'learning_rate': '5.912e-08', 'rewards/chosen': '-0.1471', 'rewards/rejected': '-0.1419', 'rewards/accuracies': '0.25', 'rewards/margins': '-0.005247', 'logps/rejected': '-1.577', 'logps/chosen': '-1.635', 'logits/rejected': '0.3225', 'logits/chosen': '0.6681', 'nll_loss': '1.661', 'log_odds_ratio': '-0.7338', 'log_odds_chosen': '-0.0736', 'epoch': '0.3636'}
{'eval_loss': '1.73', 'eval_runtime': '331.8', 'eval_samples_per_second': '0.901', 'eval_steps_per_second': '0.901', 'eval_rewards/chosen': '-0.1473', 'eval_rewards/rejected': '-0.1416', 'eval_rewards/accuracies': '0.3946', 'eval_rewards/margins': '-0.005723', 'eval_logps/rejected': '-1.573', 'eval_logps/chosen': '-1.637', 'eval_logits/rejected': '0.3217', 'eval_logits/chosen': '0.7112', 'eval_nll_loss': '1.664', 'eval_log_odds_ratio': '-0.7403', 'eval_log_odds_chosen': '-0.08069', 'epoch': '0.3636'}
{'loss': '1.718', 'grad_norm': '0.3286', 'learning_rate': '4.934e-10', 'rewards/chosen': '-0.1463', 'rewards/rejected': '-0.1407', 'rewards/accuracies': '0.425', 'rewards/margins': '-0.005628', 'logps/rejected': '-1.563', 'logps/chosen': '-1.625', 'logits/rejected': '0.3444', 'logits/chosen': '0.7127', 'nll_loss': '1.651', 'log_odds_ratio': '-0.7397', 'log_odds_chosen': '-0.07765', 'epoch': '0.4'} 
{'eval_loss': '1.73', 'eval_runtime': '331.8', 'eval_samples_per_second': '0.901', 'eval_steps_per_second': '0.901', 'eval_rewards/chosen': '-0.1473', 'eval_rewards/rejected': '-0.1416', 'eval_rewards/accuracies': '0.3913', 'eval_rewards/margins': '-0.005738', 'eval_logps/rejected': '-1.573', 'eval_logps/chosen': '-1.637', 'eval_logits/rejected': '0.3225', 'eval_logits/chosen': '0.7109', 'eval_nll_loss': '1.664', 'eval_log_odds_ratio': '-0.7404', 'eval_log_odds_chosen': '-0.08091', 'epoch': '0.4'}
{'train_runtime': '3584', 'train_samples_per_second': '0.123', 'train_steps_per_second': '0.031', 'train_loss': '1.727', 'epoch': '0.4'}                                                                
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 110/110 [59:43<00:00, 32.58s/it]

====训练完成====

【第2次训练】

调整相关参数：

beta=0.18,

learning_rate=2.5e-6

num_train_epochs=0.7,

训练日志：

python 复制代码 
root@autodl-container-4c034b95f8-320f63c6:~/autodl-tmp/codes/sft# python train_orpo_med.py 
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 339/339 [00:02<00:00, 129.88it/s]
trainable params: 40,370,176 || all params: 7,655,986,688 || trainable%: 0.5273
[RANK 0] When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments we have set it for you, but you should do it yourself in the future.
[transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151645}.
{'loss': '1.852', 'grad_norm': '0.2929', 'learning_rate': '2.25e-06', 'rewards/chosen': '-0.304', 'rewards/rejected': '-0.2908', 'rewards/accuracies': '0.225', 'rewards/margins': '-0.01313', 'logps/rejected': '-1.616', 'logps/chosen': '-1.689', 'logits/rejected': '0.2504', 'logits/chosen': '0.6192', 'nll_loss': '1.718', 'log_odds_ratio': '-0.7421', 'log_odds_chosen': '-0.08998', 'epoch': '0.03636'}
{'loss': '1.815', 'grad_norm': '0.3316', 'learning_rate': '2.485e-06', 'rewards/chosen': '-0.2977', 'rewards/rejected': '-0.277', 'rewards/accuracies': '0.425', 'rewards/margins': '-0.02074', 'logps/rejected': '-1.539', 'logps/chosen': '-1.654', 'logits/rejected': '0.2377', 'logits/chosen': '0.7499', 'nll_loss': '1.675', 'log_odds_ratio': '-0.7784', 'log_odds_chosen': '-0.1457', 'epoch': '0.07273'}
{'loss': '1.814', 'grad_norm': '0.3801', 'learning_rate': '2.434e-06', 'rewards/chosen': '-0.2979', 'rewards/rejected': '-0.2814', 'rewards/accuracies': '0.3', 'rewards/margins': '-0.01641', 'logps/rejected': '-1.564', 'logps/chosen': '-1.655', 'logits/rejected': '0.3427', 'logits/chosen': '0.6659', 'nll_loss': '1.678', 'log_odds_ratio': '-0.7573', 'log_odds_chosen': '-0.1146', 'epoch': '0.1091'}
{'loss': '1.767', 'grad_norm': '0.349', 'learning_rate': '2.348e-06', 'rewards/chosen': '-0.2902', 'rewards/rejected': '-0.2814', 'rewards/accuracies': '0.475', 'rewards/margins': '-0.008828', 'logps/rejected': '-1.563', 'logps/chosen': '-1.612', 'logits/rejected': '0.2768', 'logits/chosen': '0.7085', 'nll_loss': '1.635', 'log_odds_ratio': '-0.7318', 'log_odds_chosen': '-0.064', 'epoch': '0.1455'}
{'eval_loss': '1.801', 'eval_runtime': '331.8', 'eval_samples_per_second': '0.901', 'eval_steps_per_second': '0.901', 'eval_rewards/chosen': '-0.2954', 'eval_rewards/rejected': '-0.2839', 'eval_rewards/accuracies': '0.3913', 'eval_rewards/margins': '-0.0116', 'eval_logps/rejected': '-1.577', 'eval_logps/chosen': '-1.641', 'eval_logits/rejected': '0.3256', 'eval_logits/chosen': '0.7193', 'eval_nll_loss': '1.668', 'eval_log_odds_ratio': '-0.7409', 'eval_log_odds_chosen': '-0.08166', 'epoch': '0.1455'}
{'loss': '1.787', 'grad_norm': '0.3307', 'learning_rate': '2.23e-06', 'rewards/chosen': '-0.294', 'rewards/rejected': '-0.2849', 'rewards/accuracies': '0.375', 'rewards/margins': '-0.009056', 'logps/rejected': '-1.583', 'logps/chosen': '-1.633', 'logits/rejected': '0.3267', 'logits/chosen': '0.6863', 'nll_loss': '1.656', 'log_odds_ratio': '-0.7302', 'log_odds_chosen': '-0.06464', 'epoch': '0.1818'}
{'loss': '1.817', 'grad_norm': '0.3281', 'learning_rate': '2.083e-06', 'rewards/chosen': '-0.3', 'rewards/rejected': '-0.2902', 'rewards/accuracies': '0.375', 'rewards/margins': '-0.009752', 'logps/rejected': '-1.612', 'logps/chosen': '-1.667', 'logits/rejected': '0.1752', 'logits/chosen': '0.692', 'nll_loss': '1.685', 'log_odds_ratio': '-0.7318', 'log_odds_chosen': '-0.06729', 'epoch': '0.2182'}
{'loss': '1.742', 'grad_norm': '0.3436', 'learning_rate': '1.912e-06', 'rewards/chosen': '-0.286', 'rewards/rejected': '-0.2736', 'rewards/accuracies': '0.325', 'rewards/margins': '-0.01234', 'logps/rejected': '-1.52', 'logps/chosen': '-1.589', 'logits/rejected': '0.3022', 'logits/chosen': '0.7573', 'nll_loss': '1.609', 'log_odds_ratio': '-0.742', 'log_odds_chosen': '-0.0888', 'epoch': '0.2545'}
{'loss': '1.771', 'grad_norm': '0.3512', 'learning_rate': '1.721e-06', 'rewards/chosen': '-0.2902', 'rewards/rejected': '-0.273', 'rewards/accuracies': '0.25', 'rewards/margins': '-0.01719', 'logps/rejected': '-1.517', 'logps/chosen': '-1.612', 'logits/rejected': '0.2311', 'logits/chosen': '0.6617', 'nll_loss': '1.634', 'log_odds_ratio': '-0.7607', 'log_odds_chosen': '-0.1235', 'epoch': '0.2909'}
{'eval_loss': '1.784', 'eval_runtime': '331.7', 'eval_samples_per_second': '0.901', 'eval_steps_per_second': '0.901', 'eval_rewards/chosen': '-0.2923', 'eval_rewards/rejected': '-0.2815', 'eval_rewards/accuracies': '0.398', 'eval_rewards/margins': '-0.01079', 'eval_logps/rejected': '-1.564', 'eval_logps/chosen': '-1.624', 'eval_logits/rejected': '0.3203', 'eval_logits/chosen': '0.6972', 'eval_nll_loss': '1.651', 'eval_log_odds_ratio': '-0.738', 'eval_log_odds_chosen': '-0.07632', 'epoch': '0.2909'}
{'loss': '1.748', 'grad_norm': '0.3457', 'learning_rate': '1.516e-06', 'rewards/chosen': '-0.2869', 'rewards/rejected': '-0.2791', 'rewards/accuracies': '0.425', 'rewards/margins': '-0.007786', 'logps/rejected': '-1.55', 'logps/chosen': '-1.594', 'logits/rejected': '0.245', 'logits/chosen': '0.6811', 'nll_loss': '1.617', 'log_odds_ratio': '-0.7277', 'log_odds_chosen': '-0.05621', 'epoch': '0.3273'}
{'loss': '1.772', 'grad_norm': '0.3588', 'learning_rate': '1.304e-06', 'rewards/chosen': '-0.2904', 'rewards/rejected': '-0.2812', 'rewards/accuracies': '0.25', 'rewards/margins': '-0.009185', 'logps/rejected': '-1.562', 'logps/chosen': '-1.613', 'logits/rejected': '0.3181', 'logits/chosen': '0.6474', 'nll_loss': '1.641', 'log_odds_ratio': '-0.7292', 'log_odds_chosen': '-0.06467', 'epoch': '0.3636'}
{'loss': '1.759', 'grad_norm': '0.3422', 'learning_rate': '1.09e-06', 'rewards/chosen': '-0.288', 'rewards/rejected': '-0.2782', 'rewards/accuracies': '0.425', 'rewards/margins': '-0.00987', 'logps/rejected': '-1.545', 'logps/chosen': '-1.6', 'logits/rejected': '0.3417', 'logits/chosen': '0.6925', 'nll_loss': '1.626', 'log_odds_ratio': '-0.7349', 'log_odds_chosen': '-0.06835', 'epoch': '0.4'}
{'loss': '1.759', 'grad_norm': '0.3533', 'learning_rate': '8.801e-07', 'rewards/chosen': '-0.2879', 'rewards/rejected': '-0.275', 'rewards/accuracies': '0.4', 'rewards/margins': '-0.01296', 'logps/rejected': '-1.528', 'logps/chosen': '-1.6', 'logits/rejected': '0.2147', 'logits/chosen': '0.7227', 'nll_loss': '1.624', 'log_odds_ratio': '-0.7498', 'log_odds_chosen': '-0.09492', 'epoch': '0.4364'}
{'eval_loss': '1.767', 'eval_runtime': '331.4', 'eval_samples_per_second': '0.902', 'eval_steps_per_second': '0.902', 'eval_rewards/chosen': '-0.2893', 'eval_rewards/rejected': '-0.2793', 'eval_rewards/accuracies': '0.408', 'eval_rewards/margins': '-0.009951', 'eval_logps/rejected': '-1.552', 'eval_logps/chosen': '-1.607', 'eval_logits/rejected': '0.32', 'eval_logits/chosen': '0.6872', 'eval_nll_loss': '1.634', 'eval_log_odds_ratio': '-0.7351', 'eval_log_odds_chosen': '-0.07074', 'epoch': '0.4364'}
{'loss': '1.719', 'grad_norm': '0.4084', 'learning_rate': '6.816e-07', 'rewards/chosen': '-0.2808', 'rewards/rejected': '-0.2691', 'rewards/accuracies': '0.35', 'rewards/margins': '-0.01176', 'logps/rejected': '-1.495', 'logps/chosen': '-1.56', 'logits/rejected': '0.2761', 'logits/chosen': '0.6685', 'nll_loss': '1.585', 'log_odds_ratio': '-0.7398', 'log_odds_chosen': '-0.08312', 'epoch': '0.4727'} 
{'loss': '1.772', 'grad_norm': '0.3147', 'learning_rate': '4.997e-07', 'rewards/chosen': '-0.2905', 'rewards/rejected': '-0.2743', 'rewards/accuracies': '0.275', 'rewards/margins': '-0.01618', 'logps/rejected': '-1.524', 'logps/chosen': '-1.614', 'logits/rejected': '0.3284', 'logits/chosen': '0.6451', 'nll_loss': '1.636', 'log_odds_ratio': '-0.7574', 'log_odds_chosen': '-0.1139', 'epoch': '0.5091'}
{'loss': '1.768', 'grad_norm': '0.4449', 'learning_rate': '3.4e-07', 'rewards/chosen': '-0.2899', 'rewards/rejected': '-0.2772', 'rewards/accuracies': '0.35', 'rewards/margins': '-0.01275', 'logps/rejected': '-1.54', 'logps/chosen': '-1.611', 'logits/rejected': '0.2971', 'logits/chosen': '0.6509', 'nll_loss': '1.634', 'log_odds_ratio': '-0.7436', 'log_odds_chosen': '-0.08944', 'epoch': '0.5455'}
{'loss': '1.729', 'grad_norm': '0.3884', 'learning_rate': '2.07e-07', 'rewards/chosen': '-0.2824', 'rewards/rejected': '-0.2664', 'rewards/accuracies': '0.3', 'rewards/margins': '-0.01602', 'logps/rejected': '-1.48', 'logps/chosen': '-1.569', 'logits/rejected': '0.2604', 'logits/chosen': '0.6835', 'nll_loss': '1.592', 'log_odds_ratio': '-0.7578', 'log_odds_chosen': '-0.1158', 'epoch': '0.5818'}
{'eval_loss': '1.76', 'eval_runtime': '331.6', 'eval_samples_per_second': '0.902', 'eval_steps_per_second': '0.902', 'eval_rewards/chosen': '-0.288', 'eval_rewards/rejected': '-0.2785', 'eval_rewards/accuracies': '0.4147', 'eval_rewards/margins': '-0.009465', 'eval_logps/rejected': '-1.547', 'eval_logps/chosen': '-1.6', 'eval_logits/rejected': '0.3198', 'eval_logits/chosen': '0.6875', 'eval_nll_loss': '1.628', 'eval_log_odds_ratio': '-0.7334', 'eval_log_odds_chosen': '-0.06742', 'epoch': '0.5818'}
{'loss': '1.702', 'grad_norm': '0.3585', 'learning_rate': '1.046e-07', 'rewards/chosen': '-0.2797', 'rewards/rejected': '-0.2755', 'rewards/accuracies': '0.475', 'rewards/margins': '-0.004153', 'logps/rejected': '-1.531', 'logps/chosen': '-1.554', 'logits/rejected': '0.2413', 'logits/chosen': '0.7189', 'nll_loss': '1.574', 'log_odds_ratio': '-0.7138', 'log_odds_chosen': '-0.03262', 'epoch': '0.6182'}
{'loss': '1.761', 'grad_norm': '0.3851', 'learning_rate': '3.593e-08', 'rewards/chosen': '-0.2891', 'rewards/rejected': '-0.2769', 'rewards/accuracies': '0.35', 'rewards/margins': '-0.01219', 'logps/rejected': '-1.538', 'logps/chosen': '-1.606', 'logits/rejected': '0.2944', 'logits/chosen': '0.6761', 'nll_loss': '1.628', 'log_odds_ratio': '-0.7429', 'log_odds_chosen': '-0.08806', 'epoch': '0.6545'}
{'loss': '1.763', 'grad_norm': '0.3533', 'learning_rate': '2.946e-09', 'rewards/chosen': '-0.2884', 'rewards/rejected': '-0.2703', 'rewards/accuracies': '0.35', 'rewards/margins': '-0.01817', 'logps/rejected': '-1.501', 'logps/chosen': '-1.602', 'logits/rejected': '0.2531', 'logits/chosen': '0.6623', 'nll_loss': '1.625', 'log_odds_ratio': '-0.7647', 'log_odds_chosen': '-0.1293', 'epoch': '0.6909'}
{'eval_loss': '1.759', 'eval_runtime': '331.5', 'eval_samples_per_second': '0.902', 'eval_steps_per_second': '0.902', 'eval_rewards/chosen': '-0.2878', 'eval_rewards/rejected': '-0.2784', 'eval_rewards/accuracies': '0.4181', 'eval_rewards/margins': '-0.009388', 'eval_logps/rejected': '-1.547', 'eval_logps/chosen': '-1.599', 'eval_logits/rejected': '0.3216', 'eval_logits/chosen': '0.6876', 'eval_nll_loss': '1.627', 'eval_log_odds_ratio': '-0.7331', 'eval_log_odds_chosen': '-0.06689', 'epoch': '0.7018'}
{'train_runtime': '4487', 'train_samples_per_second': '0.172', 'train_steps_per_second': '0.043', 'train_loss': '1.768', 'epoch': '0.7018'}                                                             
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 193/193 [1:14:47<00:00, 23.25s/it]

====训练完成====
LoRA适配器保存路径：./orpo-instruct-best-adapter

结论：

  1. 验证集 accuracies 依旧远低于 0.5 随机基线

0.418 < 0.5；

beta 上调至 0.18，偏好损失权重变大，所以 accuracy 轻微上涨；

但信号强度不足以扭转模型固有的偏好

  1. eval_rewards/margins 持续负数 → reward(chosen) < reward(rejected)

模型依旧认为劣质回答更好；

  1. 指标只是小幅改善，没有持续向上突破的趋势，进入平台瓶颈；

调高 beta、增加 epoch 带来微弱增益，单纯调参已经很难进一步突破，根源不在超参，在于数据集侧。基座 Qwen 天然更容易生成短句，原始 SFT 模型本身就给 rejected 更高 log 概率。

4、问题定位

做基座基线验证：写脚本，加载【ORPO 训练前的 SFT 模型】，遍历验证集所有偏好样本，计算：logp(chosen) 与 logp(rejected)

统计：原始基座中，多少样本 logp(rejected) > logp(chosen)

两种结果对应不同对策：

1）原始基座大部分样本就更喜欢 rejected

→ 数据集先天难度极大，单纯依靠离线对齐（ORPO/DPO）很难逆转；

需要优化数据构造方式。

2）原始基座大部分样本是正常的（logp (chosen) 更高）

→ ORPO 训练发生了「对齐失效」，可以尝试更换算法、调整策略。

构造脚本（baseline_logp_analysis.py）：

python 复制代码 
import json
import torch
import pandas as pd
from transformers import AutoModelForCausalLM, AutoTokenizer

# =====================【必须修改路径】=====================
BASE_MODEL_PATH = "/root/autodl-tmp/models/Qwen2.5-7B-Instruct"
VAL_DATA_PATH = "/root/autodl-tmp/datas/rlhf/med_dpo_answer_test.jsonl"
OUTPUT_CSV = "./logp_baseline_stat.csv"

SYSTEM_PROMPT = "你是一名专业的医疗咨询助手，请严谨、客观地解答用户问题。"
MAX_SEQ_LEN = 2048
DEVICE = "cuda"
# =========================================================

def build_qwen_chat_prompt(system, user_query):
    messages = [
        {"role": "system", "content": system},
        {"role": "user", "content": user_query}
    ]
    text = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    return text

def calculate_response_log_prob(model, tokenizer, prompt, response_text):
    """
    计算回答部分token的平均对数概率 avg_logp
    """
    full_text = prompt + response_text
    inputs = tokenizer(
        full_text,
        truncation=True,
        max_length=MAX_SEQ_LEN,
        return_tensors="pt"
    ).to(DEVICE)

    prompt_inputs = tokenizer(
        prompt,
        truncation=True,
        max_length=MAX_SEQ_LEN,
        return_tensors="pt"
    ).to(DEVICE)
    prompt_len = prompt_inputs["input_ids"].shape[1]

    with torch.no_grad():
        outputs = model(**inputs)
        logits = outputs.logits
        shift_logits = logits[:, :-1, :]
        shift_labels = inputs["input_ids"][:, 1:]

        log_probs = torch.log_softmax(shift_logits, dim=-1)
        token_logp = torch.gather(log_probs, dim=-1, index=shift_labels.unsqueeze(-1)).squeeze(-1)

    # 只截取assistant回答部分token
    response_token_logp = token_logp[:, prompt_len - 1:]
    valid_tokens = response_token_logp.shape[1]

    sum_logp = float(torch.sum(response_token_logp).cpu())
    avg_logp = sum_logp / valid_tokens if valid_tokens > 0 else -float("inf")
    return sum_logp, avg_logp


if __name__ == "__main__":
    tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_PATH)
    import bitsandbytes as bnb
    from transformers import BitsAndBytesConfig

    # 4bit量化配置
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16
    )

    tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_PATH)
    model = AutoModelForCausalLM.from_pretrained(
        BASE_MODEL_PATH,
        dtype=torch.bfloat16,  # 替换过时 torch_dtype
        device_map="auto",
        quantization_config=bnb_config,  # 量化配置放这里
    )
    model.eval()
    print("✅ 原始Qwen2.5-7B-Instruct 基座加载完成")

    samples = []
    with open(VAL_DATA_PATH, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            samples.append(json.loads(line))
    print(f"✅ 验证集样本总数：{len(samples)}")

    result_list = []
    win_count = 0

    for idx, item in enumerate(samples):
        instruction = item["prompt"]
        chosen_text = item["chosen"]
        rejected_text = item["rejected"]

        prompt = build_qwen_chat_prompt(SYSTEM_PROMPT, instruction)
        sum_chosen, avg_chosen = calculate_response_log_prob(model, tokenizer, prompt, chosen_text)
        sum_rejected, avg_rejected = calculate_response_log_prob(model, tokenizer, prompt, rejected_text)

        prefer_chosen = avg_chosen > avg_rejected
        if prefer_chosen:
            win_count += 1

        result_list.append({
            "idx": idx,
            "instruction": instruction,
            "avg_logp_chosen": round(avg_chosen, 4),
            "avg_logp_rejected": round(avg_rejected, 4),
            "prefer_chosen": prefer_chosen
        })

        if (idx + 1) % 20 == 0:
            print(f"处理进度：{idx+1}/{len(samples)}")

    total = len(result_list)
    ratio = win_count / total
    print("="*60)
    print(f"基座原生偏好chosen样本：{win_count}/{total}")
    print(f"占比 = {ratio:.3f}")
    print("="*60)

    df = pd.DataFrame(result_list)
    df.to_csv(OUTPUT_CSV, index=False, encoding="utf-8-sig")
    print(f"详细结果保存至 {OUTPUT_CSV}")

【代码解读】

1）model(**inputs)单次前向

将 token 序列input_ids和attention_mask送入因果大模型前向传播。

输出包含 logits，维度batch,seq_len,vocab，代表每个位置预测下一 token 的原始分值。

model(**inputs)不会生成文本，只输出每个位置词表得分

2）model.generate (inputs)

自回归采样生成新文本

内部循环多次调用model()，只使用每轮最后一个位置 logits，不断生成下一个 token。

3）整体思路

1.关闭梯度计算，序列一次性前向传播得到所有位置原始 logits；

2.将 logits 与 input_ids 错位对齐，匹配因果模型t预测t+1规则；

3.log_softmax 得到词表对数概率，gather 提取真实 token 对应的 log 概率；

4.根据 prompt 长度切片，仅保留回答片段的对数概率；

5.求和后除以 token 数量，得到回答 token 平均对数似然。

【运行结果】

基座原生偏好chosen样本：108/299

占比 = 0.361

原始基座本身更倾向输出 rejected（劣质回答）

总共 299 条偏好样本，仅仅 108 条模型天然觉得 chosen 更好；超过 6 成样本，模型认为劣质回答概率更高。

这完美解释了你 ORPO 训练遇到的现象：

训练 loss 收敛，但是 eval 指标上不去、模型很难学到偏好，很难逆转基座先天倾向。

【问题总结】

（1）基座预训练强先验（最大因素，你的数据集痛点）

通用基座 Qwen2.5-Instruct 在万亿级文本上学到强先验：短句更容易出现。 你的数据特征： chosen（优质医疗回答）长、复杂；rejected（劣质回答）简短。 平均 logp 天然：rejected > chosen。 模型先天学到：简短文本拥有更高发生概率。

这个先验是海量预训练数据固化的，参数空间里这是一个极宽、极深的局部最优解。 想要强行翻转偏好，需要极大梯度更新，极易触发两个问题：

  • 需要非常多训练步数，容易过拟合；
  • 剧烈更新 LoRA 参数，引发分布漂移、通用能力崩塌。

（2）偏好损失的梯度能力有限

（3）只要有损失，不断训练总能反转倾向 的错误观点

数学上可行，但工程存在约束：

LoRA 参数容量有限、训练 epoch 不能无限增加、不能牺牲模型通用能力。

无限训练强行扭转先天倾向，大概率出现：

训练集偏好指标变好，域外测试泛化极差、灾难性过拟合。

（4）训练流水线缺少前置领域 SFT

你的流程：

Qwen2.5-Instruct → 直接ORPO

标准工业流程：

基座 → 医疗领域SFT → ORPO

通用基座不熟悉医疗话术体系。

在不懂领域语言的前提下，直接做偏好对齐：

模型连怎么正常输出医疗文本都不熟练，还要同时学习偏好，双重学习压力，优化难度倍增。

经过 SFT 之后：

模型先适应医疗文本风格，缩小长短句带来的先天 logp 差距，基线偏好占比提升，再跑 ORPO 事半功倍。

结论：理论上 ORPO 能够调整模型偏好，但当前基座仅 36.1% 样本天然偏好 chosen，存在巨大先天偏向。核心阻碍：通用基座预训练形成 "短句优先" 的强先验；数据集长短文本形成混淆变量，模型容易学到捷径；Hard 偏好样本下偏好损失梯度微弱。同时缺少领域 SFT 预热，进一步加大优化难度。强行训练不仅收敛缓慢，还容易过拟合、丧失泛化能力。优先均衡正负回答长度、清洗数据集，或增加领域 SFT 后再做偏好训练。

相关推荐
中微极客2 小时前
多智能体编排实战：CrewAI vs AutoGen（2026版）
大数据·网络·人工智能
葡萄城技术团队2 小时前
活字格 AI Coding——重新定义大屏开发范式（三）
人工智能
QXWZ_IA2 小时前
RTK外业成果怎么判断能不能用于正式成图？交付标准
人工智能·科技·智能硬件
骊城英雄2 小时前
Rust从入门到精通-trait
人工智能·算法·rust
阿文和她的Key2 小时前
GPT-5.6 降价后， API 账单的三层漏斗该怎么拆
人工智能·gpt·ai·chatgpt
BEOL贝尔科技2 小时前
样本安全存储中还有哪些重要因素需要考虑?如何设置样本安全?
人工智能·安全·数据分析
thesky1234563 小时前
27届大模型岗面试准备（十四）：多模态大模型 VLM——从视觉编码器到多模态推理的完整链路
人工智能·ai·大模型
一根数据线3 小时前
BIM建模效率低？试试和AI工具配合使用
人工智能·ai·3d建模·3d模型·bim·ai建模·造形家
怕浪猫3 小时前
2840亿参数只卖白菜价：DeepSeek V4 Flash 正式版上线，Agent 能力暴涨6倍
人工智能·算法
热门推荐
01如何新建文件夹？ 电脑新建文件夹的4种方法02GitHub 镜像站点032026年7月AI圈大地震：GPT-5.6被政府限制、Claude入驻Slack、Anthropic自研芯片04国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）05AI 编程 IDE 全景解析 2026：Agent 全面接管开发链路06微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新07AI科技热点日报 | 2026年07月01日082026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？092026上半年大模型全景技术解读：推理融合、Agent 爆发与多模态统一102026 年 AI 大模型 & AI 编程工具实战全总结