Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment

TL;DR

This paper introduces Cross2StrA, a novel unpaired cross-lingual image captioning framework that leverages scene graphs and syntactic trees for better semantic and syntactic alignment across languages.

Contribution

It proposes a structure-pivoted alignment approach combining scene graphs and syntactic trees with cross-modal and cross-lingual back-translation for improved captioning.

Findings

Significant improvement in caption relevancy and fluency for English-Chinese transfer.

Effective cross-modal and cross-lingual structure alignment.

Outperforms existing methods in unpaired cross-lingual captioning.

Abstract

Unpaired cross-lingual image captioning has long suffered from irrelevancy and disfluency issues, due to the inconsistencies of the semantic scene and syntax attributes during transfer. In this work, we propose to address the above problems by incorporating the scene graph (SG) structures and the syntactic constituency (SC) trees. Our captioner contains the semantic structure-guided image-to-pivot captioning and the syntactic structure-guided pivot-to-target translation, two of which are joined via pivot language. We then take the SG and SC structures as pivoting, performing cross-modal semantic structure alignment and cross-lingual syntactic structure alignment learning. We further introduce cross-lingual&cross-modal back-translation training to fully align the captioning and translation stages. Experiments on English-Chinese transfers show that our model shows great superiority in improving captioning relevancy and fluency.