Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier

TL;DR

Aya Expanse introduces a new family of large multilingual models that outperform existing models across 23 languages, setting a new state-of-the-art in multilingual NLP performance.

Contribution

The paper presents the Aya Expanse model family, combining research advancements to achieve superior multilingual performance and releasing open weights and a new multilingual evaluation dataset.

Findings

Aya Expanse models outperform leading open-weight models in multilingual tasks.

Aya Expanse 32B surpasses larger models like Llama 3.1 70B in win-rate.

The models achieve up to 76.6% win-rate on the Arena-Hard-Auto dataset.

Abstract

We introduce the Aya Expanse model family, a new generation of 8B and 32B parameter multilingual language models, aiming to address the critical challenge of developing highly performant multilingual models that match or surpass the capabilities of monolingual models. By leveraging several years of research at Cohere For AI and Cohere, including advancements in data arbitrage, multilingual preference training, and model merging, Aya Expanse sets a new state-of-the-art in multilingual performance. Our evaluations on the Arena-Hard-Auto dataset, translated into 23 languages, demonstrate that Aya Expanse 8B and 32B outperform leading open-weight models in their respective parameter classes, including Gemma 2, Qwen 2.5, and Llama 3.1, achieving up to a 76.6% win-rate. Notably, Aya Expanse 32B outperforms Llama 3.1 70B, a model with twice as many parameters, achieving a 54.0% win-rate. In this short technical report, we present extended evaluation results for the Aya Expanse model family and release their open-weights, together with a new multilingual evaluation dataset m-ArenaHard.