Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring

TL;DR

This paper introduces SCALE, a data quality assessment and selection pipeline that improves vision-language models by evaluating and selecting high-quality, well-aligned image-text data, reducing data noise and enhancing reasoning capabilities.

Contribution

We propose SCALE, a unified modality scoring framework that assesses and filters multimodal data based on alignment, clarity, and task relevance, improving VLM instruction tuning.

Findings

SCALE effectively filters high-quality data for VLM training.

Generated captions help unify multimodal data into a single text modality.

Proper data selection enhances model reasoning and robustness.

Abstract

The application of visual instruction tuning and other post-training techniques has significantly enhanced the capabilities of Large Language Models (LLMs) in visual understanding, enriching Vision-Language Models (VLMs) with more comprehensive visual language datasets. However, the effectiveness of VLMs is highly dependent on large-scale, high-quality datasets that ensure precise recognition and accurate reasoning. Two key challenges hinder progress: (1) noisy alignments between images and the corresponding text, which leads to misinterpretation, and (2) ambiguous or misleading text, which obscures visual content. To address these challenges, we propose SCALE (Single modality data quality and Cross modality Alignment Evaluation), a novel quality-driven data selection pipeline for VLM instruction tuning datasets. Specifically, SCALE integrates a cross-modality assessment framework that first assigns each data entry to its appropriate vision-language task, generates general and task-specific captions (covering scenes, objects, style, etc.), and evaluates the alignment, clarity, task rarity, text coherence, and image clarity of each entry based on the generated captions. We reveal that: (1) current unimodal quality assessment methods evaluate one modality while overlooking the rest, which can underestimate samples essential for specific tasks and discard the lower-quality instances that help build model robustness; and (2) appropriately generated image captions provide an efficient way to transfer the image-text multimodal task into a unified text modality.