Meta CLIP 2: A Worldwide Scaling Recipe

TL;DR

Meta CLIP 2 introduces a scalable training recipe for CLIP models on worldwide web data, effectively handling multilingual data and improving performance across diverse benchmarks without specialized architecture changes.

Contribution

It presents the first comprehensive recipe for training CLIP from scratch on global web-scale data, addressing multilingual challenges and surpassing previous models in zero-shot tasks.

Findings

Outperforms English-only CLIP in zero-shot ImageNet classification by 0.8%.

Achieves new state-of-the-art on multilingual benchmarks like CVQA and Babel-ImageNet.

Demonstrates effective handling of non-English data without architecture modifications.

Abstract

Contrastive Language-Image Pretraining (CLIP) is a popular foundation model, supporting from zero-shot classification, retrieval to encoders for multimodal large language models (MLLMs). Although CLIP is successfully trained on billion-scale image-text pairs from the English world, scaling CLIP's training further to learning from the worldwide web data is still challenging: (1) no curation method is available to handle data points from non-English world; (2) the English performance from existing multilingual CLIP is worse than its English-only counterpart, i.e., "curse of multilinguality" that is common in LLMs. Here, we present Meta CLIP 2, the first recipe training CLIP from scratch on worldwide web-scale image-text pairs. To generalize our findings, we conduct rigorous ablations with minimal changes that are necessary to address the above challenges and present a recipe enabling mutual benefits from English and non-English world data. In zero-shot ImageNet classification, Meta CLIP 2 ViT-H/14 surpasses its English-only counterpart by 0.8% and mSigLIP by 0.7%, and surprisingly sets new state-of-the-art without system-level confounding factors (e.g., translation, bespoke architecture changes) on multilingual benchmarks, such as CVQA with 57.4%, Babel-ImageNet with 50.2% and XM3600 with 64.3% on image-to-text retrieval.