Multimodal Pivots for Image Caption Translation

TL;DR

This paper introduces a multimodal pivoting method for image caption translation that leverages visual similarity to improve translation quality without requiring large parallel datasets.

Contribution

It proposes a novel approach using image retrieval and caption reranking based on visual similarity, reducing dependence on parallel data.

Findings

Achieved a 1 BLEU point improvement over strong baselines

Utilized CNN-based image similarity for crosslingual reranking

Demonstrated effectiveness with monolingual captioned image datasets

Abstract

We present an approach to improve statistical machine translation of image descriptions by multimodal pivots defined in visual space. The key idea is to perform image retrieval over a database of images that are captioned in the target language, and use the captions of the most similar images for crosslingual reranking of translation outputs. Our approach does not depend on the availability of large amounts of in-domain parallel data, but only relies on available large datasets of monolingually captioned images, and on state-of-the-art convolutional neural networks to compute image similarities. Our experimental evaluation shows improvements of 1 BLEU point over strong baselines.