BLIP2部署教程 - 技术栈

简单记录一下BLIP2部署的流程

主要遇到的问题还是有墙导致模型权重无法下载

环境安装

本文采用Lavis进行BLIP2的部署

1.pip 安装lavis，这里记得换一下清华源，下载会快一点

pip install salesforce-lavis

通过下方代码判断lavis库是否安装成功

python 复制代码

from lavis.models import model_zoo
print(model_zoo)

2.替换安装的库中权重文件路径

以Image Captioning任务为例

修改 lavis/configs/models/blip_caption_base_coco.yaml

该文件在python lavis库中，如果是通过git安装的，则直接更改对应git 项目文件

yaml 复制代码

model:
  arch: blip_caption
  load_finetuned: True

  pretrained: "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_capfilt_large.pth"
  finetuned: "https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP/blip_coco_caption_base.pth"

  # vit encoder
  vit_type: "base"
  vit_grad_ckpt: False
  vit_ckpt_layer: 0
  image_size: 384

将pretrained 和 finetuned 中的权重文件下载到本地，然后将其替换成权重的绝对路径

运行报错 OSError: Can't load tokenizer for 'bert-base-uncased'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'bert-base-uncased' is the correct path to a dir

直接命令行通过huggingface镜像运行脚本，从而下载对应bert模型权重
HF_ENDPOINT=https://hf-mirror.com python 测试脚本.py

总体思想就是缺少哪个模型权重文件就下载哪个，然后替换成对应绝对路径

另附Image Captioning任务测试脚本

python 复制代码

import torch
from lavis.models import load_model_and_preprocess
from PIL import Image

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# loads BLIP caption base model, with finetuned checkpoints on MSCOCO captioning dataset.
# this also loads the associated image processors
model, vis_processors, _ = load_model_and_preprocess(name="blip_caption", model_type="base_coco", is_eval=True, device=device)
# preprocess the image
# vis_processors stores image transforms for "train" and "eval" (validation / testing / inference)
raw_image = Image.open("./merlion.png").convert("RGB")

image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
# generate caption
res=model.generate({"image": image})
print(res)
# ['a large fountain spewing water into the air']