Beyond Filtering: Adaptive Image-Text Quality Enhancement for MLLM Pretraining

TL;DR

This paper introduces AITQE, a dynamic approach to improve image-text pair quality in multimodal pretraining, enhancing data utilization and model scalability without heavily modifying textual data.

Contribution

The paper proposes AITQE, a novel adaptive method that assesses and enhances image-text pair quality through text rewriting and negative sampling, improving over filter-based approaches.

Findings

AITQE outperforms existing methods on benchmark datasets.

It effectively leverages raw data and scales with data volume.

Minimal text adjustments preserve data diversity.

Abstract

Multimodal large language models (MLLMs) have made significant strides by integrating visual and textual modalities. A critical factor in training MLLMs is the quality of image-text pairs within multimodal pretraining datasets. However, $\textit {de facto}$ filter-based data quality enhancement paradigms often discard a substantial portion of high-quality image data due to inadequate semantic alignment between images and texts, leading to inefficiencies in data utilization and scalability. In this paper, we propose the Adaptive Image-Text Quality Enhancer (AITQE), a model that dynamically assesses and enhances the quality of image-text pairs. AITQE employs a text rewriting mechanism for low-quality pairs and incorporates a negative sample learning strategy to improve evaluative capabilities by integrating deliberately selected low-quality samples during training. Unlike prior approaches that significantly alter text distributions, our method minimally adjusts text to preserve data volume while enhancing quality. Experimental results demonstrate that AITQE surpasses existing methods on various benchmark, effectively leveraging raw data and scaling efficiently with increasing data volumes. We hope our work will inspire future works. The code and model are available at: https://github.com/hanhuang22/AITQE.