Universal Neural Machine Translation for Extremely Low Resource Languages

TL;DR

This paper introduces a universal neural machine translation method that leverages transfer learning and shared representations to improve translation quality for extremely low-resource languages, achieving significant BLEU score improvements.

Contribution

The paper presents a novel transfer-learning approach with shared lexical and sentence-level representations for low-resource languages, outperforming existing methods.

Findings

Achieves 23 BLEU on Romanian-English with only 6k sentences

Outperforms baseline systems using multilingual training and back-translation

Attains nearly 20 BLEU in zero-shot fine-tuning setting

Abstract

In this paper, we propose a new universal machine translation approach focusing on languages with a limited amount of parallel data. Our proposed approach utilizes a transfer-learning approach to share lexical and sentence level representations across multiple source languages into one target language. The lexical part is shared through a Universal Lexical Representation to support multilingual word-level sharing. The sentence-level sharing is represented by a model of experts from all source languages that share the source encoders with all other languages. This enables the low-resource language to utilize the lexical and sentence representations of the higher resource languages. Our approach is able to achieve 23 BLEU on Romanian-English WMT2016 using a tiny parallel corpus of 6k sentences, compared to the 18 BLEU of strong baseline system which uses multilingual training and back-translation. Furthermore, we show that the proposed approach can achieve almost 20 BLEU on the same dataset through fine-tuning a pre-trained multi-lingual system in a zero-shot setting.