Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation

TL;DR

This paper introduces a multimodal reinforcement learning approach for simultaneous machine translation that leverages visual cues to improve translation quality and reduce latency.

Contribution

It proposes a novel multimodal reinforcement learning framework integrating visual information into SiMT, exploring different strategies and demonstrating improved performance.

Findings

Visual cues enhance translation quality.

The approach maintains low latency.

Different integration strategies impact results.

Abstract

This paper addresses the problem of simultaneous machine translation (SiMT) by exploring two main concepts: (a) adaptive policies to learn a good trade-off between high translation quality and low latency; and (b) visual information to support this process by providing additional (visual) contextual information which may be available before the textual input is produced. For that, we propose a multimodal approach to simultaneous machine translation using reinforcement learning, with strategies to integrate visual and textual information in both the agent and the environment. We provide an exploration on how different types of visual information and integration strategies affect the quality and latency of simultaneous translation models, and demonstrate that visual cues lead to higher quality while keeping the latency low.