Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languages

TL;DR

This study investigates how visual context influences noisy multimodal neural machine translation from English to Indian languages, revealing that images can be redundant or beneficial depending on noise levels, and providing insights for future multimodal NMT research.

Contribution

It provides an empirical analysis of visual context effects in high-resource multilingual NMT, introducing synthetic noise and demonstrating when images aid translation quality.

Findings

Images are often redundant in high-resource settings.

Multimodal models outperform text-only models in noisy conditions.

Different visual features are optimal depending on noise level.

Abstract

Neural Machine Translation (NMT) has made remarkable progress using large-scale textual data, but the potential of incorporating multimodal inputs, especially visual information, remains underexplored in high-resource settings. While prior research has focused on using multimodal data in low-resource scenarios, this study examines how image features impact translation when added to a large-scale, pre-trained unimodal NMT system. Surprisingly, the study finds that images might be redundant in this context. Additionally, the research introduces synthetic noise to assess whether images help the model handle textual noise. Multimodal models slightly outperform text-only models in noisy settings, even when random images are used. The study's experiments translate from English to Hindi, Bengali, and Malayalam, significantly outperforming state-of-the-art benchmarks. Interestingly, the effect of visual context varies with the level of source text noise: no visual context works best for non-noisy translations, cropped image features are optimal for low noise, and full image features perform better in high-noise scenarios. This sheds light on the role of visual context, especially in noisy settings, and opens up a new research direction for Noisy Neural Machine Translation in multimodal setups. The research emphasizes the importance of combining visual and textual information to improve translation across various environments. Our code is publicly available at https://github.com/babangain/indicMMT.