Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards

TL;DR

This paper introduces a self-supervised reinforcement learning approach for unpaired cross-lingual image captioning that improves fluency and relevance without relying on pivot language translation.

Contribution

It proposes a novel self-supervised reward mechanism using monolingual data and visual semantic matching to enhance caption quality in unpaired cross-lingual settings.

Findings

Significant performance improvements over state-of-the-art methods.

Effective fluency and relevance enhancement without pivot translation.

Successful application to English and Chinese captioning tasks.

Abstract

Generating image descriptions in different languages is essential to satisfy users worldwide. However, it is prohibitively expensive to collect large-scale paired image-caption dataset for every target language which is critical for training descent image captioning models. Previous works tackle the unpaired cross-lingual image captioning problem through a pivot language, which is with the help of paired image-caption data in the pivot language and pivot-to-target machine translation models. However, such language-pivoted approach suffers from inaccuracy brought by the pivot-to-target translation, including disfluency and visual irrelevancy errors. In this paper, we propose to generate cross-lingual image captions with self-supervised rewards in the reinforcement learning framework to alleviate these two types of errors. We employ self-supervision from mono-lingual corpus in the target language to provide fluency reward, and propose a multi-level visual semantic matching model to provide both sentence-level and concept-level visual relevancy rewards. We conduct extensive experiments for unpaired cross-lingual image captioning in both English and Chinese respectively on two widely used image caption corpora. The proposed approach achieves significant performance improvement over state-of-the-art methods.