Zero-Resource Neural Machine Translation with Multi-Agent Communication Game

TL;DR

This paper introduces an interactive multimodal framework where neural translation models learn to translate by engaging in cooperative image description games, improving zero-resource translation performance.

Contribution

It presents a novel multi-agent communication game approach enabling zero-resource neural machine translation without relying on parallel corpora.

Findings

Significant improvement over state-of-the-art methods on IAPR-TC12 and Multi30K datasets.

Demonstrates the effectiveness of cooperative image description games for low-resource translation.

Shows that models can develop translation capabilities through interactive multimodal learning.

Abstract

While end-to-end neural machine translation (NMT) has achieved notable success in the past years in translating a handful of resource-rich language pairs, it still suffers from the data scarcity problem for low-resource language pairs and domains. To tackle this problem, we propose an interactive multimodal framework for zero-resource neural machine translation. Instead of being passively exposed to large amounts of parallel corpora, our learners (implemented as encoder-decoder architecture) engage in cooperative image description games, and thus develop their own image captioning or neural machine translation model from the need to communicate in order to succeed at the game. Experimental results on the IAPR-TC12 and Multi30K datasets show that the proposed learning mechanism significantly improves over the state-of-the-art methods.