Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring

TL;DR

This paper presents an unsupervised machine translation approach for German to Czech that combines word and subword models with language model rescoring to improve translation quality without using parallel data.

Contribution

It introduces a novel rescoring mechanism using a pre-trained language model and separate BPE embeddings aligned with MUSE to enhance unsupervised translation.

Findings

Improved translation fluency and accuracy demonstrated in WMT'19 results.

Effective handling of morphological richness through separate BPE training.

Rescoring with language models significantly boosts translation quality.

Abstract

This paper describes CAiRE's submission to the unsupervised machine translation track of the WMT'19 news shared task from German to Czech. We leverage a phrase-based statistical machine translation (PBSMT) model and a pre-trained language model to combine word-level neural machine translation (NMT) and subword-level NMT models without using any parallel data. We propose to solve the morphological richness problem of languages by training byte-pair encoding (BPE) embeddings for German and Czech separately, and they are aligned using MUSE (Conneau et al., 2018). To ensure the fluency and consistency of translations, a rescoring mechanism is proposed that reuses the pre-trained language model to select the translation candidates generated through beam search. Moreover, a series of pre-processing and post-processing approaches are applied to improve the quality of final translations.