WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training

TL;DR

WenLan introduces BriVL, a large-scale multi-modal pre-training model that implicitly models cross-modal correlations using a contrastive learning framework, improving vision-language understanding especially in Chinese contexts.

Contribution

The paper presents BriVL, a novel two-tower model with an advanced MoCo-based contrastive learning algorithm for large-scale Chinese multi-modal pre-training.

Findings

BriVL outperforms UNITER and CLIP on downstream tasks.

Constructed a large Chinese multi-source image-text dataset RUC-CAS-WenLan.

Effective large-scale pre-training with limited GPU resources.

Abstract

Multi-modal pre-training models have been intensively explored to bridge vision and language in recent years. However, most of them explicitly model the cross-modal interaction between image-text pairs, by assuming that there exists strong semantic correlation between the text and image modalities. Since this strong assumption is often invalid in real-world scenarios, we choose to implicitly model the cross-modal correlation for large-scale multi-modal pre-training, which is the focus of the Chinese project `WenLan' led by our team. Specifically, with the weak correlation assumption over image-text pairs, we propose a two-tower pre-training model called BriVL within the cross-modal contrastive learning framework. Unlike OpenAI CLIP that adopts a simple contrastive learning method, we devise a more advanced algorithm by adapting the latest method MoCo into the cross-modal scenario. By building a large queue-based dictionary, our BriVL can incorporate more negative samples in limited GPU resources. We further construct a large Chinese multi-source image-text dataset called RUC-CAS-WenLan for pre-training our BriVL model. Extensive experiments demonstrate that the pre-trained BriVL model outperforms both UNITER and OpenAI CLIP on various downstream tasks.