X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale

TL;DR

X-ALMA is a multilingual translation model that achieves high-quality results across 50 languages by using plug-and-play modules and a novel adaptive rejection optimization, outperforming existing models on major benchmarks.

Contribution

The paper introduces X-ALMA, a new multilingual translation model with a plug-and-play architecture and ARPO optimization, ensuring balanced high-quality translation across resource levels.

Findings

X-ALMA outperforms state-of-the-art open-source multilingual LLMs on FLORES-200 and WMT'23 datasets.

The plug-and-play module architecture prevents language conflicts during training.

ARPO optimization surpasses existing preference methods in translation quality.

Abstract

Large language models (LLMs) have achieved remarkable success across various NLP tasks with a focus on English due to English-centric pre-training and limited multilingual data. In this work, we focus on the problem of translation, and while some multilingual LLMs claim to support for hundreds of languages, models often fail to provide high-quality responses for mid- and low-resource languages, leading to imbalanced performance heavily skewed in favor of high-resource languages. We introduce **X-ALMA**, a model designed to ensure top-tier performance across 50 diverse languages, regardless of their resource levels. X-ALMA surpasses state-of-the-art open-source multilingual LLMs, such as Aya-101 and Aya-23, in every single translation direction on the FLORES-200 and WMT'23 test datasets according to COMET-22. This is achieved by plug-and-play language-specific module architecture to prevent language conflicts during training and a carefully designed training regimen with novel optimization methods to maximize the translation performance. After the final stage of training regimen, our proposed **A**daptive **R**ejection **P**reference **O**ptimization (**ARPO**) surpasses existing preference optimization methods in translation tasks.