08-PEFT源码阅读-使用指南与实战案例

08 - 使用指南与实战案例

本篇把前 7 篇理论落地为可运行代码。先总览训练-推理-合并生命周期，再分点给出安装、配置、训练、推理、合并、多适配器、QLoRA、Prefix Tuning、IA³、优化器、分布式、Hub 集成的完整案例，最后用多适配器切换图收束。

---

一、总览

PEFT 的典型工作流是固定的三段式：训练 → 保存 → 推理/合并 。get_peft_model 是唯一入口，PeftModel.from_pretrained 是推理加载入口，merge_and_unload 是合并卸载入口。

1.1 生命周期状态机

#mermaid-svg-QSJDGShIl5qG1jVh{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-QSJDGShIl5qG1jVh .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-QSJDGShIl5qG1jVh .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-QSJDGShIl5qG1jVh .error-icon{fill:#552222;}#mermaid-svg-QSJDGShIl5qG1jVh .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-QSJDGShIl5qG1jVh .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-QSJDGShIl5qG1jVh .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-QSJDGShIl5qG1jVh .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-QSJDGShIl5qG1jVh .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-QSJDGShIl5qG1jVh .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-QSJDGShIl5qG1jVh .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-QSJDGShIl5qG1jVh .marker{fill:#333333;stroke:#333333;}#mermaid-svg-QSJDGShIl5qG1jVh .marker.cross{stroke:#333333;}#mermaid-svg-QSJDGShIl5qG1jVh svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-QSJDGShIl5qG1jVh p{margin:0;}#mermaid-svg-QSJDGShIl5qG1jVh defs #statediagram-barbEnd{fill:#333333;stroke:#333333;}#mermaid-svg-QSJDGShIl5qG1jVh g.stateGroup text{fill:#9370DB;stroke:none;font-size:10px;}#mermaid-svg-QSJDGShIl5qG1jVh g.stateGroup text{fill:#333;stroke:none;font-size:10px;}#mermaid-svg-QSJDGShIl5qG1jVh g.stateGroup .state-title{font-weight:bolder;fill:#131300;}#mermaid-svg-QSJDGShIl5qG1jVh g.stateGroup rect{fill:#ECECFF;stroke:#9370DB;}#mermaid-svg-QSJDGShIl5qG1jVh g.stateGroup line{stroke:#333333;stroke-width:1;}#mermaid-svg-QSJDGShIl5qG1jVh .transition{stroke:#333333;stroke-width:1;fill:none;}#mermaid-svg-QSJDGShIl5qG1jVh .stateGroup .composit{fill:white;border-bottom:1px;}#mermaid-svg-QSJDGShIl5qG1jVh .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px;}#mermaid-svg-QSJDGShIl5qG1jVh .state-note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-QSJDGShIl5qG1jVh .state-note text{fill:black;stroke:none;font-size:10px;}#mermaid-svg-QSJDGShIl5qG1jVh .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5;}#mermaid-svg-QSJDGShIl5qG1jVh .edgeLabel .label rect{fill:#ECECFF;opacity:0.5;}#mermaid-svg-QSJDGShIl5qG1jVh .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-QSJDGShIl5qG1jVh .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-QSJDGShIl5qG1jVh .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-QSJDGShIl5qG1jVh .edgeLabel .label text{fill:#333;}#mermaid-svg-QSJDGShIl5qG1jVh .label div .edgeLabel{color:#333;}#mermaid-svg-QSJDGShIl5qG1jVh .stateLabel text{fill:#131300;font-size:10px;font-weight:bold;}#mermaid-svg-QSJDGShIl5qG1jVh .node circle.state-start{fill:#333333;stroke:#333333;}#mermaid-svg-QSJDGShIl5qG1jVh .node .fork-join{fill:#333333;stroke:#333333;}#mermaid-svg-QSJDGShIl5qG1jVh .node circle.state-end{fill:#9370DB;stroke:white;stroke-width:1.5;}#mermaid-svg-QSJDGShIl5qG1jVh .end-state-inner{fill:white;stroke-width:1.5;}#mermaid-svg-QSJDGShIl5qG1jVh .node rect{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-QSJDGShIl5qG1jVh .node polygon{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-QSJDGShIl5qG1jVh #statediagram-barbEnd{fill:#333333;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-cluster rect{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-QSJDGShIl5qG1jVh .cluster-label,#mermaid-svg-QSJDGShIl5qG1jVh .nodeLabel{color:#131300;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-cluster rect.outer{rx:5px;ry:5px;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-state .divider{stroke:#9370DB;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-state .title-state{rx:5px;ry:5px;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-cluster.statediagram-cluster .inner{fill:white;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-cluster.statediagram-cluster-alt .inner{fill:#f0f0f0;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-cluster .inner{rx:0;ry:0;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-state rect.basic{rx:5px;ry:5px;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#f0f0f0;}#mermaid-svg-QSJDGShIl5qG1jVh .note-edge{stroke-dasharray:5;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-note rect{fill:#fff5ad;stroke:#aaaa33;stroke-width:1px;rx:0;ry:0;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-note rect{fill:#fff5ad;stroke:#aaaa33;stroke-width:1px;rx:0;ry:0;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-note text{fill:black;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram-note .nodeLabel{color:black;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagram .edgeLabel{color:red;}#mermaid-svg-QSJDGShIl5qG1jVh #dependencyStart,#mermaid-svg-QSJDGShIl5qG1jVh #dependencyEnd{fill:#333333;stroke:#333333;stroke-width:1;}#mermaid-svg-QSJDGShIl5qG1jVh .statediagramTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-QSJDGShIl5qG1jVh :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} AutoModelForXxx.from_pretrained
get_peft_model(model, PeftConfig)
Trainer.train
save_pretrained("adapter/")
PeftModel.from_pretrained(base, "adapter/")
with disable_adapter()
退出上下文
merge_adapter (可 unmerge)
add_adapter + set_adapter
forward(adapter_names=...)
merge_and_unload → 裸模型
merge_and_unload
push_to_hub
基础模型
PEFT模型
训练中
已保存
推理
已禁用
已合并
多适配器
混合批
已卸载
已推送

图 1 PEFT 训练-推理-合并生命周期 ：从基础模型出发，get_peft_model 注入适配器；训练后可保存/推送 Hub；推理可禁用/合并/多适配器/混合批；merge_and_unload 产出裸模型退出 PEFT 域。

---

二、分述

2.1 安装

# 稳定版
pip install peft

# 从源码（开发版）
git clone https://github.com/huggingface/peft.git
cd peft
pip install -e .

# 测试依赖
pip install ".[test]"

# 量化训练额外需要
pip install bitsandbytes      # QLoRA 主力
pip install gptqmodel autoawq # GPTQ/AWQ

2.2 案例 1：Causal LM + LoRA（最小训练样例）

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from peft import LoraConfig, TaskType, get_peft_model

device = "cuda" if torch.cuda.is_available() else "cpu"
model_id = "Qwen/Qwen2.5-3B-Instruct"

model = AutoModelForCausalLM.from_pretrained(model_id, device_map=device)
tokenizer = AutoTokenizer.from_pretrained(model_id)

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],  # 可省略, 用模型默认
)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
# trainable params: 3,686,400 || all params: 3,089,625,088 || trainable%: 0.1193

# 接入 transformers Trainer 训练
training_args = TrainingArguments(output_dir="./qwen-lora", num_train_epochs=3,
                                  per_device_train_batch_size=4, learning_rate=2e-4)
trainer = Trainer(model=model, args=training_args, train_dataset=your_dataset, tokenizer=tokenizer)
trainer.train()
model.save_pretrained("./qwen-lora")

2.3 案例 2：推理加载与合并

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", device_map="auto")
model = PeftModel.from_pretrained(base, "./qwen-lora")     # 加载适配器
model = model.merge_and_unload()                           # 合并并卸载为裸模型

inputs = tokenizer("Preheat the oven to 350 degrees", return_tensors="pt").to("cuda")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))

不合并推理（保留适配器、支持热切换）：

model = PeftModel.from_pretrained(base, "./qwen-lora", is_trainable=False)
# 推理时临时禁用适配器
with model.disable_adapter():
    print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))

2.4 案例 3：多适配器管理与切换

from peft import LoraConfig, get_peft_model

model = get_peft_model(base, LoraConfig(r=8, target_modules=["q_proj","v_proj"], task_type="CAUSAL_LM"))
# 新增第二个适配器
model.add_adapter("math", LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","k_proj","v_proj","o_proj"]))
model.add_adapter("code", LoraConfig(r=32, target_modules=["q_proj","v_proj"]))

model.set_adapter("math")      # 切到 math 适配器
out_math = model.generate(**inputs)
model.set_adapter("code")      # 零开销切到 code
out_code = model.generate(**inputs)

# 加权融合成新适配器
model.add_weighted_adapter(
    adapters=["math", "code"], weights=[0.6, 0.4],
    adapter_name="math_code", combination_type="svd",
)
model.set_adapter("math_code")

# 删除
model.delete_adapter("code")

2.5 案例 4：QLoRA 4bit 训练（消费级 GPU）

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf", quantization_config=bnb_config, device_map="auto",
)
model = prepare_model_for_kbit_training(model)   # 关键预处理
model = get_peft_model(model, LoraConfig(
    r=64, lora_alpha=16, lora_dropout=0.1,
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
    task_type="CAUSAL_LM",
))
model.print_trainable_parameters()
# trainable params: ~40M || all params: ~3.5B(含4bit) || trainable%: ~1.1%
# 7B 模型可在 16GB 单卡上训练

2.6 案例 5：Sequence Classification + IA³

from transformers import AutoModelForSequenceClassification
from peft import IA3Config, TaskType, get_peft_model

model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased", num_labels=2,
)
peft_config = IA3Config(
    task_type=TaskType.SEQ_CLS,
    target_modules=["query", "value", "key", "dense"],
    feedforward_modules=["dense"],   # 必须是 target_modules 子集
    modules_to_save=["classifier"],  # 分类头需训练保存
)
model = get_peft_model(model, peft_config)

2.7 案例 6：Seq2Seq + Prefix Tuning

from transformers import AutoModelForSeq2SeqLM
from peft import PrefixTuningConfig, TaskType, get_peft_model

model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
peft_config = PrefixTuningConfig(
    task_type=TaskType.SEQ_2_SEQ_LM,
    num_virtual_tokens=20,
    prefix_projection=True,
    encoder_hidden_size=512,
)
model = get_peft_model(model, peft_config)

2.8 案例 7：LoRA+ 与 LoraFA 优化器

from peft.optimizers import create_loraplus_optimizer, create_lorafa_optimizer

# LoRA+: 给 lora_B 用更大学习率 (论文推荐 lr_ratio=16)
optimizer = create_loraplus_optimizer(
    model, optimizer_cls=torch.optim.AdamW,
    lr=2e-4, loraplus_lr_ratio=16,
)

# LoraFA: 冻结 lora_A, 用 A 的伪逆投影 lora_B 梯度
optimizer = create_lorafa_optimizer(model, r=16, lora_alpha=32, lr=2e-4)

LoRA+ 在 lorafa.py/loraplus.py 内构建多参数组：lora_A 用基础 lr、lora_B 用 lr·ratio、嵌入用低 lr；LoraFA 直接冻结 A 并改写优化器步骤。

2.9 案例 8：自定义 target_modules 与图像分类

import torch
from transformers import AutoModelForImageClassification
from peft import LoraConfig, get_peft_model

model = AutoModelForImageClassification.from_pretrained("google/vit-base-patch16-224")
# 用正则匹配所有 attention 的 q/v 投影
peft_config = LoraConfig(
    r=8, lora_alpha=16,
    target_modules=r".*\.query|.*\.value",   # regex
    lora_dropout=0.1,
    modules_to_save=["classifier"],          # 分类头训练
)
model = get_peft_model(model, peft_config)

2.10 案例 9：混合批推理（每样本不同适配器）

# 4 个样本, 前 2 个用 "math", 后 2 个用 "code"
inputs = tokenizer(["math q1", "math q2", "code q1", "code q2"], return_tensors="pt", padding=True).to("cuda")
adapter_names = ["math", "math", "code", "code"]
outputs = model(**inputs, adapter_names=adapter_names)

adapter_names 经 PeftModel._enable_peft_forward_hooks 注入到各 LoraLayer，由 _mixed_batch_forward 按 sample 路由------单次 forward 同时服务多适配器，省去批切分。

2.11 案例 10：DeepSpeed ZeRO-3 分布式训练

accelerate config 选用 ZeRO-3 + CPU offload，生成 `examples/sft/configs/deepspeed_config_z3_qlora.yaml`(file:///workspace/examples/sft/configs/deepspeed_config_z3_qlora.yaml)：

compute_environment: LOCAL_MACHINE
distributed_type: DEEPSPEED
deepspeed_config:
  deepspeed_config_file: configs/deepspeed_config_z3_qlora.yaml
  zero3_init_flag: true

启动：

accelerate launch --config_file configs/deepspeed_config_z3_qlora.yaml \
    examples/sft/train.py --model_id Qwen/Qwen2.5-7B --lora_r 16 --qlora

FSDP 类似，参考 `examples/sft/configs/fsdp_config_qlora.yaml`(file:///workspace/examples/sft/configs/fsdp_config_qlora.yaml)。注意 FSDP 下合并需在 fsdp.summon_full_params 上下文里做。

2.12 案例 11：保存到 Hub 与加载

# 训练后推送
model.push_to_hub("your-username/qwen-lora", token="hf_xxx")
# 仅推配置
model.peft_config["default"].push_to_hub("your-username/qwen-lora-config")

# 他人加载
from peft import PeftModel
model = PeftModel.from_pretrained(base, "your-username/qwen-lora")

2.13 多适配器切换示意图

#mermaid-svg-OokvJU4Zw45l2HuW{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-OokvJU4Zw45l2HuW .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-OokvJU4Zw45l2HuW .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-OokvJU4Zw45l2HuW .error-icon{fill:#552222;}#mermaid-svg-OokvJU4Zw45l2HuW .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-OokvJU4Zw45l2HuW .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-OokvJU4Zw45l2HuW .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-OokvJU4Zw45l2HuW .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-OokvJU4Zw45l2HuW .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-OokvJU4Zw45l2HuW .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-OokvJU4Zw45l2HuW .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-OokvJU4Zw45l2HuW .marker{fill:#333333;stroke:#333333;}#mermaid-svg-OokvJU4Zw45l2HuW .marker.cross{stroke:#333333;}#mermaid-svg-OokvJU4Zw45l2HuW svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-OokvJU4Zw45l2HuW p{margin:0;}#mermaid-svg-OokvJU4Zw45l2HuW .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-OokvJU4Zw45l2HuW .cluster-label text{fill:#333;}#mermaid-svg-OokvJU4Zw45l2HuW .cluster-label span{color:#333;}#mermaid-svg-OokvJU4Zw45l2HuW .cluster-label span p{background-color:transparent;}#mermaid-svg-OokvJU4Zw45l2HuW .label text,#mermaid-svg-OokvJU4Zw45l2HuW span{fill:#333;color:#333;}#mermaid-svg-OokvJU4Zw45l2HuW .node rect,#mermaid-svg-OokvJU4Zw45l2HuW .node circle,#mermaid-svg-OokvJU4Zw45l2HuW .node ellipse,#mermaid-svg-OokvJU4Zw45l2HuW .node polygon,#mermaid-svg-OokvJU4Zw45l2HuW .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-OokvJU4Zw45l2HuW .rough-node .label text,#mermaid-svg-OokvJU4Zw45l2HuW .node .label text,#mermaid-svg-OokvJU4Zw45l2HuW .image-shape .label,#mermaid-svg-OokvJU4Zw45l2HuW .icon-shape .label{text-anchor:middle;}#mermaid-svg-OokvJU4Zw45l2HuW .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-OokvJU4Zw45l2HuW .rough-node .label,#mermaid-svg-OokvJU4Zw45l2HuW .node .label,#mermaid-svg-OokvJU4Zw45l2HuW .image-shape .label,#mermaid-svg-OokvJU4Zw45l2HuW .icon-shape .label{text-align:center;}#mermaid-svg-OokvJU4Zw45l2HuW .node.clickable{cursor:pointer;}#mermaid-svg-OokvJU4Zw45l2HuW .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-OokvJU4Zw45l2HuW .arrowheadPath{fill:#333333;}#mermaid-svg-OokvJU4Zw45l2HuW .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-OokvJU4Zw45l2HuW .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-OokvJU4Zw45l2HuW .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-OokvJU4Zw45l2HuW .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-OokvJU4Zw45l2HuW .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-OokvJU4Zw45l2HuW .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-OokvJU4Zw45l2HuW .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-OokvJU4Zw45l2HuW .cluster text{fill:#333;}#mermaid-svg-OokvJU4Zw45l2HuW .cluster span{color:#333;}#mermaid-svg-OokvJU4Zw45l2HuW div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-OokvJU4Zw45l2HuW .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-OokvJU4Zw45l2HuW rect.text{fill:none;stroke-width:0;}#mermaid-svg-OokvJU4Zw45l2HuW .icon-shape,#mermaid-svg-OokvJU4Zw45l2HuW .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-OokvJU4Zw45l2HuW .icon-shape p,#mermaid-svg-OokvJU4Zw45l2HuW .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-OokvJU4Zw45l2HuW .icon-shape rect,#mermaid-svg-OokvJU4Zw45l2HuW .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-OokvJU4Zw45l2HuW .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-OokvJU4Zw45l2HuW .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-OokvJU4Zw45l2HuW :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} ModuleDict (每层独立)
基础模型 (冻结)
替换
替换
替换
替换
激活
激活
激活
q_proj
v_proj
k_proj
o_proj
LoraLayer
LoraLayer
LoraLayer
LoraLayer
default: r=8
math: r=16
code: r=32
math_code: svd融合
set_adapter('math')
set_adapter('code')
set_adapter('math_code')

图 2 多适配器切换示意 ：每层 LoraLayer 内部用 ModuleDict 持有全部适配器权重；set_adapter 只翻转 _active_adapter 与 requires_grad，零权重迁移。add_weighted_adapter 产出融合适配器并加入池。

---

三、小结

PEFT 的使用模型极简：get_peft_model(model, PeftConfig) 一行注入，save_pretrained/from_pretrained/merge_and_unload 三板斧走完生命周期 。进阶能力（多适配器、混合批、QLoRA、Prefix Tuning、LoRA+、分布式）都是在这条主线上叠加配置与少量额外调用。结合前 7 篇的原理理解，这些案例的可调试性与可扩展性都已透------比如想换成 IA³，只需把 LoraConfig 换成 IA3Config，其余代码不动。

下一篇 09-总结与最佳实践 收束全局，给出选型决策树、性能权衡、常见陷阱与 FAQ。