M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training

TL;DR

M3P is a unified multitask pre-training model that learns universal, multilingual, and multimodal representations, improving cross-lingual image retrieval performance, especially for non-English languages.

Contribution

The paper introduces M3P, a novel framework combining multilingual and multimodal pre-training with a code-switch strategy for better cross-lingual multimodal understanding.

Findings

Achieves state-of-the-art results on non-English image retrieval tasks.

Effectively aligns images with multiple languages in a shared semantic space.

Demonstrates the benefit of multitask pre-training for multilingual multimodal tasks.

Abstract

We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training. Our goal is to learn universal representations that can map objects occurred in different modalities or texts expressed in different languages into a common semantic space. In addition, to explicitly encourage fine-grained alignment between images and non-English languages, we also propose Multimodal Code-switched Training (MCT) to combine monolingual pre-training and multimodal pre-training via a code-switch strategy. Experiments are performed on the multilingual image retrieval task across two benchmark datasets, including MSCOCO and Multi30K. M3P can achieve comparable results for English and new state-of-the-art results for non-English languages.