Large-scale Bilingual Language-Image Contrastive Learning

TL;DR

This paper presents KELIP, a large-scale bilingual Korean-English multimodal model trained on 1.1 billion image-text pairs, demonstrating effective training schemes and exploring cultural and semantic aspects of multimodal learning.

Contribution

The work introduces a large-scale bilingual multimodal dataset and a model with simple training schemes, revealing insights into cross-lingual and cultural semantic learning.

Findings

Training schemes like MAE and multi-crop augmentation improve performance.

Multimodal models can learn cross-lingual relations without explicit cross-lingual training.

KELIP captures cultural differences in visual semantics.

Abstract

This paper is a technical report to share our experience and findings building a Korean and English bilingual multimodal model. While many of the multimodal datasets focus on English and multilingual multimodal research uses machine-translated texts, employing such machine-translated texts is limited to describing unique expressions, cultural information, and proper noun in languages other than English. In this work, we collect 1.1 billion image-text pairs (708 million Korean and 476 million English) and train a bilingual multimodal model named KELIP. We introduce simple yet effective training schemes, including MAE pre-training and multi-crop augmentation. Extensive experiments demonstrate that a model trained with such training schemes shows competitive performance in both languages. Moreover, we discuss multimodal-related research questions: 1) strong augmentation-based methods can distract the model from learning proper multimodal relations; 2) training multimodal model without cross-lingual relation can learn the relation via visual semantics; 3) our bilingual KELIP can capture cultural differences of visual semantics for the same meaning of words; 4) a large-scale multimodal model can be used for multimodal feature analogy. We hope that this work will provide helpful experience and findings for future research. We provide an open-source pre-trained KELIP.