RealSyn: An Effective and Scalable Multimodal Interleaved Document Transformation Paradigm

TL;DR

RealSyn is a large-scale dataset of realistic and synthetic image-text pairs designed to improve contrastive vision-language models, demonstrating state-of-the-art results across multiple benchmarks.

Contribution

The paper introduces RealSyn, a novel scalable dataset with a hierarchical retrieval and augmentation pipeline, significantly enhancing contrastive learning performance.

Findings

Models trained on RealSyn outperform existing datasets in downstream tasks.

RealSyn improves zero-shot transfer and robustness.

The dataset demonstrates scalable benefits for vision-language pre-training.

Abstract

After pre-training on extensive image-text pairs, Contrastive Language-Image Pre-training (CLIP) demonstrates promising performance on a wide variety of benchmarks. However, a substantial volume of multimodal interleaved documents remains underutilized for contrastive vision-language representation learning. To fully leverage these unpaired documents, we initially establish a Real-World Data Extraction pipeline to extract high-quality images and texts. Then we design a hierarchical retrieval method to efficiently associate each image with multiple semantically relevant realistic texts. To further enhance fine-grained visual information, we propose an image semantic augmented generation module for synthetic text production. Furthermore, we employ a semantic balance sampling strategy to improve dataset diversity, enabling better learning of long-tail concepts. Based on these innovations, we construct RealSyn, a dataset combining realistic and synthetic texts, available in three scales: 15M, 30M, and 100M. We compare our dataset with other widely used datasets of equivalent scale for CLIP training. Models pre-trained on RealSyn consistently achieve state-of-the-art performance across various downstream tasks, including linear probe, zero-shot transfer, zero-shot robustness, and zero-shot retrieval. Furthermore, extensive experiments confirm that RealSyn significantly enhances contrastive vision-language representation learning and demonstrates robust scalability. To facilitate future research, the RealSyn dataset and pretrained model weights are released at https://github.com/deepglint/RealSyn.