Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets

TL;DR

This study investigates the role of visual information in multimodal machine translation using authentic datasets, revealing that visuals generally aid translation and are supplementary rather than essential.

Contribution

The paper demonstrates the importance of visual modality in real-world datasets and challenges prior conclusions based on limited, artificial datasets.

Findings

Visual modality improves translation in authentic datasets

Translation quality depends on text-visual alignment

Visual information can be substituted without major loss

Abstract

Recent research in the field of multimodal machine translation (MMT) has indicated that the visual modality is either dispensable or offers only marginal advantages. However, most of these conclusions are drawn from the analysis of experimental results based on a limited set of bilingual sentence-image pairs, such as Multi30k. In these kinds of datasets, the content of one bilingual parallel sentence pair must be well represented by a manually annotated image, which is different from the real-world translation scenario. In this work, we adhere to the universal multimodal machine translation framework proposed by Tang et al. (2022). This approach allows us to delve into the impact of the visual modality on translation efficacy by leveraging real-world translation datasets. Through a comprehensive exploration via probing tasks, we find that the visual modality proves advantageous for the majority of authentic translation datasets. Notably, the translation performance primarily hinges on the alignment and coherence between textual and visual contents. Furthermore, our results suggest that visual information serves a supplementary role in multimodal translation and can be substituted.