VALHALLA: Visual Hallucination for Machine Translation

TL;DR

VALHALLA introduces a novel multimodal machine translation approach that hallucines visual representations from source text, enabling effective translation without requiring paired images during inference, thus broadening real-world applicability.

Contribution

The paper proposes a visual hallucination framework for machine translation that predicts visual features from text, eliminating the need for paired images at inference time.

Findings

Outperforms text-only baselines on multiple datasets

Achieves competitive results compared to multimodal methods with paired images

Demonstrates robustness across diverse language pairs

Abstract

Designing better machine translation systems by considering auxiliary inputs such as images has attracted much attention in recent years. While existing methods show promising performance over the conventional text-only translation systems, they typically require paired text and image as input during inference, which limits their applicability to real-world scenarios. In this paper, we introduce a visual hallucination framework, called VALHALLA, which requires only source sentences at inference time and instead uses hallucinated visual representations for multimodal machine translation. In particular, given a source sentence an autoregressive hallucination transformer is used to predict a discrete visual representation from the input text, and the combined text and hallucinated representations are utilized to obtain the target translation. We train the hallucination transformer jointly with the translation transformer using standard backpropagation with cross-entropy losses while being guided by an additional loss that encourages consistency between predictions using either ground-truth or hallucinated visual representations. Extensive experiments on three standard translation datasets with a diverse set of language pairs demonstrate the effectiveness of our approach over both text-only baselines and state-of-the-art methods. Project page: http://www.svcl.ucsd.edu/projects/valhalla.