Unsupervised Transfer Learning in Multilingual Neural Machine Translation with Cross-Lingual Word Embeddings

TL;DR

This paper explores unsupervised methods for adding new languages to multilingual neural machine translation systems using cross-lingual embeddings, achieving competitive translation quality without direct bilingual data.

Contribution

It introduces a novel approach combining cross-lingual embeddings and blind decoding for unsupervised language addition in NMT, with practical adaptation via non-iterative backtranslation.

Findings

Blind decoding achieves 36.4 BLEU for Portuguese-English.

Autoencoder training yields up to 28 BLEU with noise.

Non-iterative backtranslation attains 34.6 BLEU, close to supervised models.

Abstract

In this work we look into adding a new language to a multilingual NMT system in an unsupervised fashion. Under the utilization of pre-trained cross-lingual word embeddings we seek to exploit a language independent multilingual sentence representation to easily generalize to a new language. While using cross-lingual embeddings for word lookup we decode from a yet entirely unseen source language in a process we call blind decoding. Blindly decoding from Portuguese using a basesystem containing several Romance languages we achieve scores of 36.4 BLEU for Portuguese-English and 12.8 BLEU for Russian-English. In an attempt to train the mapping from the encoder sentence representation to a new target language we use our model as an autoencoder. Merely training to translate from Portuguese to Portuguese while freezing the encoder we achieve 26 BLEU on English-Portuguese, and up to 28 BLEU when adding artificial noise to the input. Lastly we explore a more practical adaptation approach through non-iterative backtranslation, exploiting our model's ability to produce high quality translations through blind decoding. This yields us up to 34.6 BLEU on English-Portuguese, attaining near parity with a model adapted on real bilingual data.