部署Kimi-VL-A3B-Instruct视频推理

部署Kimi-VL-A3B-Instruct视频推理

契机

⚙ 最近国内AI公司月之暗面推出了Kimi-VL 开源视觉模型。模型参数16.4B,但是推理时候激活参数2.8B。看了huggingface主页的Full comparison,在多项Benchmark 的时候都展示出了不俗的实力。由于业务中使用了qwen-vl-2.5-7B,所以测试下Kimi-VL-A3B-Instruct是否在更小的部署资源中拥有更好的表现。

moonshotai/Kimi-VL-A3B-Instruct · Hugging Face

安装

bash 复制代码
conda create --name Kimi-VL python=3.11.0
conda activate Kimi-VL
git clone https://github.com/MoonshotAI/Kimi-VL.git
pip install -r requirements.txt
#额外安装视频推理所需依赖
pip install opencv-python
#flash-attn的安装会访问github,如果网络没有魔法,建议尝试其他办法
pip install -U flash-attn --use-pep517 --no-build-isolation

#模型下载
moonshotai/Kimi-VL-A3B-Instruct

#运行期间用以下命令查看显卡资源,按照0.5s刷新一次
nvidia-smi -l 0.5

图片推理测试

代码

bash 复制代码
from tracemalloc import start
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor
import torch
import time

model_path = "moonshotai/Kimi-VL-A3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
    attn_implementation="flash_attention_2"
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

while True:
    time.sleep(1)
    start = time.time()
    image_path = "Kimi-VL/figures/demo1.png"
    image = Image.open(image_path)
    messages = [
        {"role": "user", "content": [{"type": "image", "image": image_path}, {"type": "text", "text": "20字描述图片"}]}
    ]
    text = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
    inputs = processor(images=image, text=text, return_tensors="pt", padding=True, truncation=True).to(model.device)
    generated_ids = model.generate(**inputs, max_new_tokens=512)
    generated_ids_trimmed = [
        out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
    ]
    response = processor.batch_decode(
        generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
    )[0]
    print(response)
    print(time.time() - start)

输入输出

bash 复制代码
多次输出如下:
城市天际线,高楼林立,交通繁忙,夕阳余晖。
1.8491706848144531
城市天际线,高楼林立,交通繁忙,夕阳余晖。
1.2099909782409668
城市天际线,高楼林立,交通繁忙,夕阳余晖。
1.1886250972747803
城市天际线,高楼林立,交通繁忙,夕阳余晖。
1.1849017143249512
城市天际线,高楼林立,交通繁忙,夕阳余晖。
1.1930198669433594

资源占用

视频推理测试

代码

bash 复制代码
import time
import cv2
import argparse
import torch
import os 
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor

# Function to extract frames from video, save them, and return paths
def extract_frames(video_path, save_dir, target_fps=1, max_frames=1):
    """Extracts up to max_frames from a video file at target FPS, saves them, and returns their paths."""
    frame_paths = []
    # Create save directory if it doesn't exist
    os.makedirs(save_dir, exist_ok=True)

    try:
        cap = cv2.VideoCapture(video_path)
        if not cap.isOpened():
            print(f"Error: Could not open video file: {video_path}")
            return frame_paths

        video_fps = cap.get(cv2.CAP_PROP_FPS)
        if video_fps <= 0:
            print(f"Warning: Could not get video FPS for {video_path}. Assuming 30 FPS.")
            video_fps = 30

        frame_interval = max(1, int(round(video_fps / target_fps)))
        frame_count = 0
        saved_frame_count = 0

        while cap.isOpened() and saved_frame_count < max_frames:
            ret, frame = cap.read()
            if not ret:
                break

            if frame_count % frame_interval == 0:
                # Convert BGR (OpenCV) to RGB (PIL)
                frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                img = Image.fromarray(frame_rgb)

                # Save frame to disk
                frame_filename = f"frame_{saved_frame_count:04d}.png"
                frame_filepath = os.path.join(save_dir, frame_filename)
                try:
                    img.save(frame_filepath)
                    frame_paths.append(frame_filepath)
                    saved_frame_count += 1
                except Exception as save_e:
                    print(f"Error saving frame {frame_filepath}: {save_e}")
                    # Optionally continue to next frame or break

            frame_count += 1

        cap.release()
        print(f"Extracted and saved {len(frame_paths)} frames to {save_dir} from {video_path} at ~{target_fps} FPS.")

    except Exception as e:
        print(f"An error occurred during frame extraction: {e}")
        if 'cap' in locals() and cap.isOpened():
            cap.release()

    return frame_paths

# --- Argument Parsing ---
# parser = argparse.ArgumentParser(description="Describe video frame(s) using Kimi-VL model.")
# parser.add_argument("video_path", help="Path to the input video file.")
# parser.add_argument("--max_frames", type=int, default=1, help="Maximum number of frames to extract and process.")
# parser.add_argument("--save_dir", default="./test-frames", help="Directory to save extracted frames.")
# parser.add_argument("--prompt", default="Describe this video", help="Text prompt for the model.")
# parser.add_argument("--target_fps", type=float, default=1.0, help="Target frames per second for extraction.")
# args = parser.parse_args()

# --- Frame Extraction & Saving ---
# extracted_frame_paths = extract_frames(args.video_path, args.save_dir, target_fps=args.target_fps, max_frames=args.max_frames)
extracted_frame_paths = extract_frames("test.mp4", "./test_kimi_frames", target_fps=4, max_frames=40)

if not extracted_frame_paths:
    print("No frames were extracted or saved. Exiting.")
    exit()

# --- Model Loading ---
model_path = "moonshotai/Kimi-VL-A3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
    attn_implementation="flash_attention_2"
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

# --- Prepare Input for Model --- 
# Load images from the saved paths
loaded_images = []
for frame_path in extracted_frame_paths:
    if os.path.exists(frame_path):
        loaded_images.append(Image.open(frame_path))
    else:
        print(f"Warning: Saved frame path not found: {frame_path}")

print(f'Loaded {len(loaded_images)} images for processing.')

# Construct the messages list
content = []
for frame_path in extracted_frame_paths:
    content.append({"type": "image", "image": frame_path})
# content.append({"type": "text", "text": args.prompt})
content.append({"type": "text", "text": '描述视频'})

messages = [
    {"role": "user", "content": content}
]

print('Messages:')
print(messages)

# Process text and image paths
text = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

# --- Model Generation ---
while True:
    time.sleep(1)
    try:
        start =time.time()
        # Use the loaded images in the processor call
        inputs = processor(
            images=loaded_images[0] if len(loaded_images) == 1 else loaded_images, 
            text=text, 
            return_tensors="pt", 
            padding=True, 
            truncation=True
        ).to(model.device)
        
        generated_ids = model.generate(**inputs, max_new_tokens=512)
        generated_ids_trimmed = [
            out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
        ]
        response = processor.batch_decode(
            generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
        )[0]
        print("--- Model Response ---")
        print(response)
        print(time.time()-start)
    except Exception as e:
        print(f"An error occurred during model generation: {e}")

输入输出

bash 复制代码
7s视频,每秒4帧
Extracted and saved 26 frames to ./test_kimi_frames from test.mp4 at ~4 FPS

--- Model Response ---
视频开始时展示了一片宁静的海洋和蓝天,海面上波光粼粼,天空中散布着几朵白云。随后,一个世界地图的图像逐渐浮现,地图上标注了不同国家的边界。地图上方有一行日文文字,解释了蓝色代表墨卡托投影法,红色代表实际大小。地图下方有一个按钮,显示为"Show Mercator Size",暗示可以通过点击按钮来切换地图的显示方式。整个画面给人一种宁静而开阔的感觉,仿佛置身于大海之中,同时通过地图的对比,引发对地理和地图投影的思考。
13.988295316696167
--- Model Response ---
视频开始时展示了一片宁静的海洋和蓝天,海面上波光粼粼,天空中散布着几朵白云。随后,一个世界地图的图像逐渐浮现,地图上标注了不同国家的边界。地图上方有一行日文文字,解释了蓝色代表墨卡托投影法,红色代表实际大小。地图下方有一个按钮,显示为"Show Mercator Size",暗示可以通过点击按钮来切换地图的显示方式。整个画面给人一种宁静而开阔的感觉,仿佛置身于大海之中,同时通过地图的对比,引发对地理和地图投影的思考。
13.980722665786743
--- Model Response ---
视频开始时展示了一片宁静的海洋和蓝天,海面上波光粼粼,天空中散布着几朵白云。随后,一个世界地图的图像逐渐浮现,地图上标注了不同国家的边界。地图上方有一行日文文字,解释了蓝色代表墨卡托投影法,红色代表实际大小。地图下方有一个按钮,显示为"Show Mercator Size",暗示可以通过点击按钮来切换地图的显示方式。整个画面给人一种宁静而开阔的感觉,仿佛置身于大海之中,同时通过地图的对比,引发对地理和地图投影的思考。
13.964439153671265
--- Model Response ---
视频开始时展示了一片宁静的海洋和蓝天,海面上波光粼粼,天空中散布着几朵白云。随后,一个世界地图的图像逐渐浮现,地图上标注了不同国家的边界。地图上方有一行日文文字,解释了蓝色代表墨卡托投影法,红色代表实际大小。地图下方有一个按钮,显示为"Show Mercator Size",暗示可以通过点击按钮来切换地图的显示方式。整个画面给人一种宁静而开阔的感觉,仿佛置身于大海之中,同时通过地图的对比,引发对地理和地图投影的思考。
13.99526834487915

资源占用

总结

  • 这个显存占用也不低啊
  • 纯记录

写到最后

相关推荐
karmueo4625 分钟前
视频序列和射频信号多模态融合算法Fusion-Vital解读
算法·音视频·多模态
写代码的小球3 小时前
求模运算符c
算法
大千AI助手7 小时前
DTW模版匹配:弹性对齐的时间序列相似度度量算法
人工智能·算法·机器学习·数据挖掘·模版匹配·dtw模版匹配
YuTaoShao8 小时前
【LeetCode 热题 100】48. 旋转图像——转置+水平翻转
java·算法·leetcode·职场和发展
生态遥感监测笔记8 小时前
GEE利用已有土地利用数据选取样本点并进行分类
人工智能·算法·机器学习·分类·数据挖掘
Tony沈哲9 小时前
macOS 上为 Compose Desktop 构建跨架构图像处理 dylib:OpenCV + libraw + libheif 实践指南
opencv·算法
刘海东刘海东9 小时前
结构型智能科技的关键可行性——信息型智能向结构型智能的转变(修改提纲)
人工智能·算法·机器学习
pumpkin8451410 小时前
Rust 调用 C 函数的 FFI
c语言·算法·rust
挺菜的10 小时前
【算法刷题记录(简单题)003】统计大写字母个数(java代码实现)
java·数据结构·算法