AIGC笔记--SVD中UNet加载预训练权重

1--加载方式

  1. 加载全参数(.ckpt)

  2. 加载LoRA(.safetensors)

2--简单实例

python 复制代码
import sys
sys.path.append("/mnt/dolphinfs/hdd_pool/docker/user/hadoop-waimai-aigc/liujinfu/Codes/v3d-vgen-motion")

import torch
from peft import LoraConfig
from safetensors import safe_open

from svd.models.i2v_svd_unet import UNetSpatioTemporalConditionModel
from svd.utils.util import zero_rank_print

if __name__ == "__main__":

    pretrained_model_path = "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-waimai-aigc/liujinfu/Codes/svd_models/models/stable-video-diffusion-img2vid-xt"
    unet = UNetSpatioTemporalConditionModel.from_pretrained(pretrained_model_path, subfolder = "unet")

    # resume_checkpoint_path = "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-waimai-aigc/liujinfu/Codes/v3d-vgen-motion/results/outputs_motionlora_realRota_0603_1024_stride5/test-0-2024-06-03T14-31-30/checkpoints/checkpoint-500.safetensors"
    resume_checkpoint_path = "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-waimai-aigc/liujinfu/Codes/v3d-vgen-motion/results/outputs_motionFull_realRota_0529_stride5/test-0-2024-05-29T10-04-34/checkpoints/checkpoint-step-5000.ckpt"

    # Load pretrained unet weights
    if resume_checkpoint_path.endswith(".ckpt"):
        zero_rank_print(f"resume from checkpoint: {resume_checkpoint_path}")
        resume_checkpoint = torch.load(resume_checkpoint_path, map_location="cpu")
        # resume dit parameters
        print(f'resume_checkpoint keys: {resume_checkpoint.keys()}')
        state_dict = resume_checkpoint["state_dict"]
        m, u = unet.load_state_dict(state_dict, strict=False)
        zero_rank_print(f"dit missing keys: {len(m)}, unexpected keys: {len(u)}")
        assert len(u) == 0
        # resume global step
        resume_global_step = False
        if "global_step" in resume_checkpoint and resume_global_step:
            zero_rank_print(f"resume global_step: {resume_checkpoint['global_step']}")
            global_step = resume_checkpoint['global_step']    
            
    elif resume_checkpoint_path.endswith(".safetensors"):

        unet_lora_config = LoraConfig(
            r = 64, 
            lora_alpha = 64, # scaling = lora_alpha / r
            init_lora_weights = "gaussian", 
            target_modules = ["to_q","to_k","to_v","to_out.0"],
            lora_dropout = 0.1
        )
        unet.add_adapter(unet_lora_config)

        zero_rank_print(f"resume from safetensors: {resume_checkpoint_path}")
        
        state_dict = {}
        with safe_open(resume_checkpoint_path, framework="pt", device="cpu") as f:
            for key in f.keys():
                key_ = key.replace('unet.', '').replace('.weight', '')
                state_dict[key_] = f.get_tensor(key)

                u = 0
                try:
                    unet.get_submodule(key_+'.default').state_dict()['weight'].data.copy_(state_dict[key_])
                except:
                    u += 1
        assert u == 0, "resume unet params failed"

    print("All Done!")
相关推荐
狼爷12 小时前
看懂了历史,为什么你反而更怕 AI
aigc
Csvn12 小时前
同一个问题问 100 遍,钱就扣 100 次?精确缓存让它只扣一次(C02)
人工智能·aigc·agent
ZhengEnCi16 小时前
AI 规则越多越费 Token:无关规则让每次回答多耗 46% 的规则体检提示词
aigc·ai编程·claude
字节跳动开源17 小时前
OpenViking 3D理解模型上线轴测八视图及PMI解析
人工智能·开源·aigc
Nturmoils18 小时前
文件明明在本地,为什么自动验收仍说没交?我的 SoduWork 使用记录
aigc
小虎AI生活19 小时前
最近爆火的 Chrome 新 skill 装进 WorkBuddy,AI 写代码起飞了
aigc·ai编程
Dawson Zhu21 小时前
【分布式训练⑧】通信优化:NCCL、Ring AllReduce与计算通信重叠
人工智能·语言模型·架构·aigc·agi
倔强的石头_1 天前
SoduWork 实测:630 字老员工叮嘱才能做对的活,新人一句话 + 一个插件全做对了
aigc
threerocks1 天前
Seedream 5.5 vs GPT Image 2.6:15 道题实测,同一份提示词,不挑图
aigc
wangruofeng1 天前
AI Models 1.1.0 发布,查大模型价格、对比模型和找 API 地址
前端·aigc·ai编程