Contrastive Learning for Many-to-many Multilingual Neural Machine Translation

TL;DR

This paper introduces mRASP2, a contrastive learning-based training method that enhances many-to-many multilingual translation, especially improving non-English directions by aligning cross-language representations.

Contribution

It proposes a novel contrastive learning approach combined with data augmentation to unify multilingual translation models, significantly improving non-English translation quality.

Findings

Outperforms existing models on English-centric directions.

Achieves 10+ BLEU improvement on non-English directions.

Comparable or better than mBART on multiple translation tasks.

Abstract

Existing multilingual machine translation approaches mainly focus on English-centric directions, while the non-English directions still lag behind. In this work, we aim to build a many-to-many translation system with an emphasis on the quality of non-English language directions. Our intuition is based on the hypothesis that a universal cross-language representation leads to better multilingual translation performance. To this end, we propose mRASP2, a training method to obtain a single unified multilingual translation model. mRASP2 is empowered by two techniques: a) a contrastive learning scheme to close the gap among representations of different languages, and b) data augmentation on both multiple parallel and monolingual data to further align token representations. For English-centric directions, mRASP2 outperforms existing best unified model and achieves competitive or even better performance than the pre-trained and fine-tuned model mBART on tens of WMT's translation directions. For non-English directions, mRASP2 achieves an improvement of average 10+ BLEU compared with the multilingual Transformer baseline. Code, data and trained models are available at https://github.com/PANXiao1994/mRASP2.