【Qwen】DataArguments说明

DataArguments

Holds all configuration options for data loading and preprocessing in Qwen-VL fine-tuning. Passed as data_args after parsing from the command line (e.g. via HfArgumentParser) and used by make_supervised_data_module to build the dataset and collator.


Attributes

Name Type Default Description
dataset_use str "" Comma-separated dataset names or paths. Resolved via data_list() to get annotation_path and data_path for LazySupervisedDataset.
data_flatten bool False If True, use FlattenedDataCollatorForSupervisedDataset and packed sequences; otherwise use DataCollatorForSupervisedDataset.
data_packing bool False If True, enable sequence packing in the dataset (_get_packed_item).
base_interval int 2 Base interval used in packing or flattening (exact meaning depends on data_list / collator implementation).
max_pixels int 28 * 28 * 576 Maximum number of pixels (e.g. H * W) for an image. Written to the image processor's size["longest_edge"] / max_pixels.
min_pixels int 28 * 28 * 16 Minimum number of pixels for an image. Written to the image processor's size["shortest_edge"] / min_pixels.
video_max_frames int or None 8 Maximum number of sampled frames per video (used by video processor if present).
video_min_frames int or None 4 Minimum number of sampled frames per video.
video_max_pixels int 1024 * 28 * 28 Maximum total pixels for video frames. Set on the video processor when available.
video_min_pixels int 256 * 28 * 28 Minimum total pixels for video frames.
video_fps float 2 Frames per second used when sampling video.

Usage

Parsed together with ModelArguments and TrainingArguments in the training script:

python 复制代码
parser = transformers.HfArgumentParser(
    (ModelArguments, DataArguments, TrainingArguments)
)
model_args, data_args, training_args = parser.parse_args_into_dataclasses()

data_module = make_supervised_data_module(processor, data_args=data_args)

Command-line example:

bash 复制代码
python qwenvl/train/train_qwen.py \
    --dataset_use "path/to/annotations.json" \
    --data_flatten True \
    --max_pixels 50176 \
    --min_pixels 784

Note

  • DataArguments is defined in qwenvl/train/argument.py and is a dataclass. The parsed instance is typically named data_args in the training pipeline.
  • The image processor's pixel limits are updated in update_processor_pixels(processor, data_args) using max_pixels and min_pixels.
相关推荐
汤姆宝库8 小时前
RoPE旋转位置编码的实现与长上下文外推
python·深度学习·机器学习
tellmewhoisi8 小时前
深度学习0:数学知识:(L1正则化和L2正则化)
人工智能·深度学习
货拉拉技术8 小时前
视觉底座:货拉拉OCR的进阶之路
算法
__underline__8 小时前
冷门Python知识点005篇:list.sort( )的排序稳定性,是用0字节额外内存实现的
python·算法
回眸&啤酒鸭9 小时前
【回眸】Hy4 模型效果展示与能力评测大纲
大数据·人工智能·机器学习
AI小怪兽10 小时前
你的视觉基础模型会说“植物语“吗?面向目标检测的视觉基础模型植物语法
人工智能·深度学习·机器学习
峯58010 小时前
机器学习05,用 scikit-learn 做加州房价预测的完整入门工程
人工智能·机器学习·scikit-learn
伞伞悦读10 小时前
【第86期】分类指标与错误成本对照的实验与排查
大数据·人工智能·机器学习
风落青芜11 小时前
数据结构:堆的概念与实现
c语言·开发语言·数据结构·算法
AI源码11 小时前
基于机器学习的优衣库销售预测与决策系统
人工智能·机器学习