The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation

TL;DR

The FLORES-101 benchmark provides a high-quality, multilingual evaluation dataset with 3001 sentences across 101 languages, enabling better assessment of low-resource and multilingual machine translation models.

Contribution

We introduce FLORES-101, a comprehensive, professionally translated benchmark dataset for low-resource and multilingual machine translation evaluation.

Findings

Enables evaluation of many-to-many multilingual translation systems

Provides high-quality, multilingual aligned translations for 101 languages

Fosters progress in low-resource machine translation research

Abstract

One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource languages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the FLORES-101 evaluation benchmark, consisting of 3001 sentences extracted from English Wikipedia and covering a variety of different topics and domains. These sentences have been translated in 101 languages by professional translators through a carefully controlled process. The resulting dataset enables better assessment of model quality on the long tail of low-resource languages, including the evaluation of many-to-many multilingual translation systems, as all translations are multilingually aligned. By publicly releasing such a high-quality and high-coverage dataset, we hope to foster progress in the machine translation community and beyond.