Sieve: Multimodal Dataset Pruning Using Image Captioning Models

TL;DR

This paper introduces Sieve, a novel dataset pruning method using image captioning models and semantic similarity, which improves the quality of vision-language datasets and enhances downstream task performance.

Contribution

Sieve leverages synthetic captions and semantic similarity to better evaluate image-text alignment, overcoming limitations of existing CLIPScore-based pruning methods.

Findings

Outperforms CLIPScore by 2.6% and 1.7% on downstream tasks.

Achieves 2.7% and 4.5% improvements on retrieval tasks.

Enhances dataset quality leading to better VLM performance.

Abstract

Vision-Language Models (VLMs) are pretrained on large, diverse, and noisy web-crawled datasets. This underscores the critical need for dataset pruning, as the quality of these datasets is strongly correlated with the performance of VLMs on downstream tasks. Using CLIPScore from a pretrained model to only train models using highly-aligned samples is one of the most successful methods for pruning. We argue that this approach suffers from multiple limitations including: false positives and negatives due to CLIP's pretraining on noisy labels. We propose a pruning signal, Sieve, that employs synthetic captions generated by image-captioning models pretrained on small, diverse, and well-aligned image-text pairs to evaluate the alignment of noisy image-text pairs. To bridge the gap between the limited diversity of generated captions and the high diversity of alternative text (alt-text), we estimate the semantic textual similarity in the embedding space of a language model pretrained on unlabeled text corpus. Using DataComp, a multimodal dataset filtering benchmark, when evaluating on 38 downstream tasks, our pruning approach, surpasses CLIPScore by 2.6\% and 1.7\% on medium and large scale respectively. In addition, on retrieval tasks, Sieve leads to a significant improvement of 2.7% and 4.5% on medium and large scale respectively.