CapsFusion: Rethinking Image-Text Data at Scale

TL;DR

CapsFusion is a novel framework that enhances multimodal pretraining data by refining web-based and synthetic image-text captions, leading to improved model performance, efficiency, and knowledge retention.

Contribution

It introduces CapsFusion, a framework using large language models to improve caption quality, addressing scalability and knowledge loss issues in multimodal training.

Findings

CapsFusion captions outperform existing ones in benchmark scores.

CapsFusion reduces training computation by 11-16 times.

CapsFusion enhances world knowledge and scalability of models.

Abstract

Large multimodal models demonstrate remarkable generalist ability to perform diverse multimodal tasks in a zero-shot manner. Large-scale web-based image-text pairs contribute fundamentally to this success, but suffer from excessive noise. Recent studies use alternative captions synthesized by captioning models and have achieved notable benchmark performance. However, our experiments reveal significant Scalability Deficiency and World Knowledge Loss issues in models trained with synthetic captions, which have been largely obscured by their initial benchmark success. Upon closer examination, we identify the root cause as the overly-simplified language structure and lack of knowledge details in existing synthetic captions. To provide higher-quality and more scalable multimodal pretraining data, we propose CapsFusion, an advanced framework that leverages large language models to consolidate and refine information from both web-based image-text pairs and synthetic captions. Extensive experiments show that CapsFusion captions exhibit remarkable all-round superiority over existing captions in terms of model performance (e.g., 18.8 and 18.3 improvements in CIDEr score on COCO and NoCaps), sample efficiency (requiring 11-16 times less computation than baselines), world knowledge depth, and scalability. These effectiveness, efficiency and scalability advantages position CapsFusion as a promising candidate for future scaling of LMM training.