Exploiting Curriculum Learning in Unsupervised Neural Machine Translation

TL;DR

This paper introduces a curriculum learning approach for unsupervised neural machine translation that leverages pseudo data quality assessment to improve translation accuracy and training efficiency.

Contribution

It proposes a novel curriculum learning framework that uses cross-lingual embeddings and model-based quality scores to prioritize pseudo data during training.

Findings

Achieves consistent improvements on multiple translation benchmarks.

Faster convergence compared to standard unsupervised NMT.

Enhances translation quality by focusing on higher-quality pseudo data.

Abstract

Back-translation (BT) has become one of the de facto components in unsupervised neural machine translation (UNMT), and it explicitly makes UNMT have translation ability. However, all the pseudo bi-texts generated by BT are treated equally as clean data during optimization without considering the quality diversity, leading to slow convergence and limited translation performance. To address this problem, we propose a curriculum learning method to gradually utilize pseudo bi-texts based on their quality from multiple granularities. Specifically, we first apply cross-lingual word embedding to calculate the potential translation difficulty (quality) for the monolingual sentences. Then, the sentences are fed into UNMT from easy to hard batch by batch. Furthermore, considering the quality of sentences/tokens in a particular batch are also diverse, we further adopt the model itself to calculate the fine-grained quality scores, which are served as learning factors to balance the contributions of different parts when computing loss and encourage the UNMT model to focus on pseudo data with higher quality. Experimental results on WMT 14 En-Fr, WMT 16 En-De, WMT 16 En-Ro, and LDC En-Zh translation tasks demonstrate that the proposed method achieves consistent improvements with faster convergence speed.