Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translation

TL;DR

This paper introduces Bi-ACL, a novel framework that leverages monolingual data and bilingual dictionaries to address data imbalance and representation degeneration in multilingual neural machine translation, improving performance across languages.

Contribution

The paper proposes Bi-ACL, a new approach combining bidirectional autoencoders and contrastive learning with curriculum sampling to enhance multilingual translation models.

Findings

Improves translation quality for low-resource languages.

Enhances model robustness in zero-shot translation scenarios.

Effective in both high-resource and low-resource language pairs.

Abstract

Despite advances in multilingual neural machine translation (MNMT), we argue that there are still two major challenges in this area: data imbalance and representation degeneration. The data imbalance problem refers to the imbalance in the amount of parallel corpora for all language pairs, especially for long-tail languages (i.e., very low-resource languages). The representation degeneration problem refers to the problem of encoded tokens tending to appear only in a small subspace of the full space available to the MNMT model. To solve these two issues, we propose Bi-ACL, a framework that uses only target-side monolingual data and a bilingual dictionary to improve the performance of the MNMT model. We define two modules, named bidirectional autoencoder and bidirectional contrastive learning, which we combine with an online constrained beam search and a curriculum learning sampling strategy. Extensive experiments show that our proposed method is more effective both in long-tail languages and in high-resource languages. We also demonstrate that our approach is capable of transferring knowledge between domains and languages in zero-shot scenarios.