Multimodal Machine Translation with Visual Scene Graph Pruning

TL;DR

This paper introduces a novel multimodal machine translation approach that uses visual scene graph pruning to reduce noise and redundancy in visual data, improving translation quality.

Contribution

It proposes a new method leveraging language scene graph information to prune visual scene graphs, addressing visual data redundancy in multimodal translation.

Findings

PSG outperforms state-of-the-art methods in experiments.

Visual scene graph pruning reduces noise in translation tasks.

Ablation studies confirm the effectiveness of pruning.

Abstract

Multimodal machine translation (MMT) seeks to address the challenges posed by linguistic polysemy and ambiguity in translation tasks by incorporating visual information. A key bottleneck in current MMT research is the effective utilization of visual data. Previous approaches have focused on extracting global or region-level image features and using attention or gating mechanisms for multimodal information fusion. However, these methods have not adequately tackled the issue of visual information redundancy in MMT, nor have they proposed effective solutions. In this paper, we introduce a novel approach--multimodal machine translation with visual Scene Graph Pruning (PSG), which leverages language scene graph information to guide the pruning of redundant nodes in visual scene graphs, thereby reducing noise in downstream translation tasks. Through extensive comparative experiments with state-of-the-art methods and ablation studies, we demonstrate the effectiveness of the PSG model. Our results also highlight the promising potential of visual information pruning in advancing the field of MMT.