Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning

TL;DR

The paper introduces EHAT, a novel transformer-based framework that models local and global cross-lingual and cross-modal relationships for image captioning in multiple languages, improving performance on multilingual datasets.

Contribution

It proposes a heterogeneous attention transformer with specialized modules to better capture cross-domain relationships in cross-lingual image captioning tasks.

Findings

EHAT outperforms existing monolingual methods on MSCOCO dataset.

The model effectively generates accurate captions in English and Chinese.

Heterogeneous attention mechanisms enhance cross-lingual and cross-modal alignment.

Abstract

Cross-lingual image captioning is a challenging task that requires addressing both cross-lingual and cross-modal obstacles in multimedia analysis. The crucial issue in this task is to model the global and the local matching between the image and different languages. Existing cross-modal embedding methods based on the transformer architecture oversee the local matching between the image region and monolingual words, especially when dealing with diverse languages. To overcome these limitations, we propose an Embedded Heterogeneous Attention Transformer (EHAT) to establish cross-domain relationships and local correspondences between images and different languages by using a heterogeneous network. EHAT comprises Masked Heterogeneous Cross-attention (MHCA), Heterogeneous Attention Reasoning Network (HARN), and Heterogeneous Co-attention (HCA). The HARN serves as the core network and it captures cross-domain relationships by leveraging visual bounding box representation features to connect word features from two languages and to learn heterogeneous maps. MHCA and HCA facilitate cross-domain integration in the encoder through specialized heterogeneous attention mechanisms, enabling a single model to generate captions in two languages. We evaluate our approach on the MSCOCO dataset to generate captions in English and Chinese, two languages that exhibit significant differences in their language families. The experimental results demonstrate the superior performance of our method compared to existing advanced monolingual methods. Our proposed EHAT framework effectively addresses the challenges of cross-lingual image captioning, paving the way for improved multilingual image analysis and understanding.