A Shared Task on Bandit Learning for Machine Translation

TL;DR

This paper presents a shared task on bandit learning for machine translation, focusing on learning from weak user feedback rather than traditional references, with results from various architectures and protocols.

Contribution

It introduces a novel shared task on bandit learning for machine translation, including setup, data, metrics, and evaluation of multiple systems.

Findings

Different architectures show varied effectiveness in bandit learning

Weak feedback can be used effectively for machine translation improvement

Shared task setup facilitates benchmarking in this research area

Abstract

We introduce and describe the results of a novel shared task on bandit learning for machine translation. The task was organized jointly by Amazon and Heidelberg University for the first time at the Second Conference on Machine Translation (WMT 2017). The goal of the task is to encourage research on learning machine translation from weak user feedback instead of human references or post-edits. On each of a sequence of rounds, a machine translation system is required to propose a translation for an input, and receives a real-valued estimate of the quality of the proposed translation for learning. This paper describes the shared task's learning and evaluation setup, using services hosted on Amazon Web Services (AWS), the data and evaluation metrics, and the results of various machine translation architectures and learning protocols.