Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation

TL;DR

This paper introduces a new framework and dataset for multimodal machine translation that leverages large-scale non-triple data, improving translation quality in realistic scenarios and outperforming existing models.

Contribution

It proposes a 2/3-Triplet framework utilizing monolingual and parallel text data, and constructs the EMMT dataset for more practical evaluation of MMT systems.

Findings

Significant performance improvement with non-triple data.

Outperforms state-of-the-art models on benchmarks.

Better suitability for real-world applications.

Abstract

Multimodal machine translation (MMT) aims to improve translation quality by incorporating information from other modalities, such as vision. Previous MMT systems mainly focus on better access and use of visual information and tend to validate their methods on image-related datasets. These studies face two challenges. First, they can only utilize triple data (bilingual texts with images), which is scarce; second, current benchmarks are relatively restricted and do not correspond to realistic scenarios. Therefore, this paper correspondingly establishes new methods and new datasets for MMT. First, we propose a framework 2/3-Triplet with two new approaches to enhance MMT by utilizing large-scale non-triple data: monolingual image-text data and parallel text-only data. Second, we construct an English-Chinese {e}-commercial {m}ulti{m}odal {t}ranslation dataset (including training and testing), named EMMT, where its test set is carefully selected as some words are ambiguous and shall be translated mistakenly without the help of images. Experiments show that our method is more suitable for real-world scenarios and can significantly improve translation performance by using more non-triple data. In addition, our model also rivals various SOTA models in conventional multimodal translation benchmarks.