Scaling Pre-training to One Hundred Billion Data for Vision Language Models

TL;DR

This study explores the effects of pre-training vision-language models on an unprecedented 100 billion web examples, revealing insights into performance saturation, cultural diversity, and multilinguality, emphasizing the importance of large-scale data for inclusive AI.

Contribution

It provides the first comprehensive empirical analysis of vision-language models trained on 100 billion examples, highlighting the impact on diversity, multilinguality, and benchmark performance.

Findings

Performance saturates on standard benchmarks at this scale.

Cultural diversity benefits significantly from large-scale web data.

Low-resource languages see improved performance with larger datasets.

Abstract

We provide an empirical investigation of the potential of pre-training vision-language models on an unprecedented scale: 100 billion examples. We find that model performance tends to saturate at this scale on many common Western-centric classification and retrieval benchmarks, such as COCO Captions. Nevertheless, tasks of cultural diversity achieve more substantial gains from the 100-billion scale web data, thanks to its coverage of long-tail concepts. Furthermore, we analyze the model's multilinguality and show gains in low-resource languages as well. In addition, we observe that reducing the size of the pretraining dataset via quality filters like using CLIP, typically used to enhance performance, may inadvertently reduce the cultural diversity represented even in large-scale datasets. Our results highlight that while traditional benchmarks may not benefit significantly from scaling noisy, raw web data to 100 billion examples, this data scale is vital for building truly inclusive multimodal systems.