Image Pivoting for Learning Multilingual Multimodal Representations

TL;DR

This paper introduces a model that learns shared multilingual multimodal representations by using images as pivots, enabling improved image and sentence matching across languages without requiring parallel data.

Contribution

It proposes a novel pivot-based approach with a new ranking loss for multilingual multimodal learning, achieving state-of-the-art results.

Findings

State-of-the-art performance on German-English image-description ranking

Effective multilingual image understanding without parallel data

Improved semantic textual similarity in image descriptions

Abstract

In this paper we propose a model to learn multimodal multilingual representations for matching images and sentences in different languages, with the aim of advancing multilingual versions of image search and image understanding. Our model learns a common representation for images and their descriptions in two different languages (which need not be parallel) by considering the image as a pivot between two languages. We introduce a new pairwise ranking loss function which can handle both symmetric and asymmetric similarity between the two modalities. We evaluate our models on image-description ranking for German and English, and on semantic textual similarity of image descriptions in English. In both cases we achieve state-of-the-art performance.