Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task

TL;DR

This paper introduces a novel method for learning multimodal multilingual embeddings that improve cross-modal retrieval tasks by aligning images and captions across languages, achieving state-of-the-art results.

Contribution

It combines existing objectives with a new alignment technique for multilingual embeddings, enhancing cross-modal retrieval performance.

Findings

Achieves state-of-the-art in text-to-image retrieval

Improves caption-caption similarity accuracy

Demonstrates effectiveness on Multi30k and Microsoft-COCO datasets

Abstract

In this paper, we propose a new approach to learn multimodal multilingual embeddings for matching images and their relevant captions in two languages. We combine two existing objective functions to make images and captions close in a joint embedding space while adapting the alignment of word embeddings between existing languages in our model. We show that our approach enables better generalization, achieving state-of-the-art performance in text-to-image and image-to-text retrieval task, and caption-caption similarity task. Two multimodal multilingual datasets are used for evaluation: Multi30k with German and English captions and Microsoft-COCO with English and Japanese captions.