【Qwen】DataArguments说明

DataArguments

Holds all configuration options for data loading and preprocessing in Qwen-VL fine-tuning. Passed as data_args after parsing from the command line (e.g. via HfArgumentParser) and used by make_supervised_data_module to build the dataset and collator.

Attributes

Name	Type	Default	Description
`dataset_use`	`str`	`""`	Comma-separated dataset names or paths. Resolved via `data_list()` to get `annotation_path` and `data_path` for `LazySupervisedDataset`.
`data_flatten`	`bool`	`False`	If `True`, use `FlattenedDataCollatorForSupervisedDataset` and packed sequences; otherwise use `DataCollatorForSupervisedDataset`.
`data_packing`	`bool`	`False`	If `True`, enable sequence packing in the dataset (`_get_packed_item`).
`base_interval`	`int`	`2`	Base interval used in packing or flattening (exact meaning depends on `data_list` / collator implementation).
`max_pixels`	`int`	`28 * 28 * 576`	Maximum number of pixels (e.g. `H * W`) for an image. Written to the image processor's `size["longest_edge"]` / `max_pixels`.
`min_pixels`	`int`	`28 * 28 * 16`	Minimum number of pixels for an image. Written to the image processor's `size["shortest_edge"]` / `min_pixels`.
`video_max_frames`	`int` or `None`	`8`	Maximum number of sampled frames per video (used by video processor if present).
`video_min_frames`	`int` or `None`	`4`	Minimum number of sampled frames per video.
`video_max_pixels`	`int`	`1024 * 28 * 28`	Maximum total pixels for video frames. Set on the video processor when available.
`video_min_pixels`	`int`	`256 * 28 * 28`	Minimum total pixels for video frames.
`video_fps`	`float`	`2`	Frames per second used when sampling video.

Usage

Parsed together with ModelArguments and TrainingArguments in the training script:

python 复制代码

parser = transformers.HfArgumentParser(
    (ModelArguments, DataArguments, TrainingArguments)
)
model_args, data_args, training_args = parser.parse_args_into_dataclasses()

data_module = make_supervised_data_module(processor, data_args=data_args)

Command-line example:

bash 复制代码

python qwenvl/train/train_qwen.py \
    --dataset_use "path/to/annotations.json" \
    --data_flatten True \
    --max_pixels 50176 \
    --min_pixels 784

Note

DataArguments is defined in qwenvl/train/argument.py and is a dataclass. The parsed instance is typically named data_args in the training pipeline.
The image processor's pixel limits are updated in update_processor_pixels(processor, data_args) using max_pixels and min_pixels.