Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters

TL;DR

This paper introduces a new filtering framework using fine-tuned multimodal language models that surpasses existing methods like CLIPScore in selecting high-quality image-text data, thereby enhancing downstream model performance.

Contribution

The authors develop a novel MLM-based filtering method with four metrics, outperforming CLIPScore and providing a versatile, drop-in replacement for improved data quality in multimodal tasks.

Findings

MLM filters outperform CLIPScore in data quality metrics.

Filtered data improves performance on foundation models and downstream tasks.

The approach generalizes across different models and tasks.

Abstract

We propose a novel framework for filtering image-text data by leveraging fine-tuned Multimodal Language Models (MLMs). Our approach outperforms predominant filtering methods (e.g., CLIPScore) via integrating the recent advances in MLMs. We design four distinct yet complementary metrics to holistically measure the quality of image-text data. A new pipeline is established to construct high-quality instruction data for fine-tuning MLMs as data filters. Comparing with CLIPScore, our MLM filters produce more precise and comprehensive scores that directly improve the quality of filtered data and boost the performance of pre-trained models. We achieve significant improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2) and various downstream tasks. Our MLM filter can generalize to different models and tasks, and be used as a drop-in replacement for CLIPScore. An additional ablation study is provided to verify our design choices for the MLM filter.