A Visual Attention Grounding Neural Model for Multimodal Machine Translation

TL;DR

This paper presents a novel multimodal machine translation model that integrates visual attention grounding with textual translation, achieving state-of-the-art results on multiple datasets and introducing a new multilingual product description dataset.

Contribution

The paper introduces a visual attention grounding mechanism within a multimodal translation model and a new multilingual dataset for real-world applications.

Findings

Achieves competitive state-of-the-art results on Multi30K and Ambiguous COCO datasets.

Outperforms existing methods significantly on the new product description dataset.

Demonstrates the effectiveness of visual grounding in improving translation quality.

Abstract

We introduce a novel multimodal machine translation model that utilizes parallel visual and textual information. Our model jointly optimizes the learning of a shared visual-language embedding and a translator. The model leverages a visual attention grounding mechanism that links the visual semantics with the corresponding textual semantics. Our approach achieves competitive state-of-the-art results on the Multi30K and the Ambiguous COCO datasets. We also collected a new multilingual multimodal product description dataset to simulate a real-world international online shopping scenario. On this dataset, our visual attention grounding model outperforms other methods by a large margin.