Finetune LLaVA on Custom Datasets

Dataset Format

Convert your data to a JSON file of a List of all samples. Sample metadata should contain id (a unique identifier), image (the path to the image), and conversations (the conversation data between human and AI).

A sample JSON for finetuning LLaVA for generating tag-style captions for Stable Diffusion:

json 复制代码
[
  {
    "id": "997bb945-628d-4724-b370-b84de974a19f",
    "image": "part-000001/997bb945-628d-4724-b370-b84de974a19f.jpg",
    "conversations": [
      {
        "from": "human",
        "value": "<image>\nWrite a prompt for Stable Diffusion to generate this image."
      },
      {
        "from": "gpt",
        "value": "a beautiful painting of chernobyl by nekro, pascal blanche, john harris, greg rutkowski, sin jong hun, moebius, simon stalenhag. in style of cg art. ray tracing. cel shading. hyper detailed. realistic. ue 5. maya. octane render. "
      },
    ]
  },
  ...
]

Command

If you have a limited task-specific data, we recommend finetuning from LLaVA checkpoints with LoRA following this script.

If the amount of the task-specific data is sufficient, you can also finetune from LLaVA checkpoints with full-model finetuning following this script.

You may need to adjust the hyperparameters to fit each specific dataset and your hardware constraint.

相关推荐
山顶夕景3 天前
【ECCV2026】流式视频理解Thinking in Streaming Video
音视频·vlm·视频理解·多模态理解·流式视频
AlfredZhao8 天前
OCR、VLM、ASR 到底有什么区别?一文讲清
ocr·asr·vlm
Felaim1 个月前
LightNav-0论文精读
vlm·vln·通才导航·rvq·双通道指向
山顶夕景1 个月前
【Jev】VideoJev: System One Model for Video Understanding and Interaction
vlm·视频分类·视频理解·多模态理解·jev
山顶夕景1 个月前
【VLM】DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
多模态·kv·vlm·infra
山顶夕景1 个月前
【VLA】Qwen-VLA:VLM+DiT统一多模态理解与连续机器人动作生成
机器人·多模态·扩散模型·具身智能·vla·dit·vlm
一直在努力的小宁2 个月前
【阅读笔记】具身智能的真机数采,到了分水岭
人工智能·深度学习·机器学习·agent·具身智能·vlm·vln
Robot_Nav2 个月前
ICRA 2025|LocoVLM:视觉与语言驱动的通用足式运动策略自适应【文献解读】
四足机器人·vlm·locovlm
一直在努力的小宁2 个月前
【阅读笔记】具身操作的数采方案概览
后端·json·restful·具身智能·vla·vlm
一颗小树x2 个月前
vLLM大模型推理:Jetson AGX Thor 的 GPU 共享内存清理实战
jetson·vllm·vlm·gpu内存清理