Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss

TL;DR

This paper critiques current multilingual benchmarks for not accurately measuring multilingual proficiency and proposes round-trip translation as a more effective evaluation method, introducing the LiT benchmark.

Contribution

It introduces round-trip translation as a novel evaluation method for multilingual models and presents the LiT benchmark for realistic multilingual assessment.

Findings

Round-trip translation correlates highly (0.94) with user ratings on LMArena.

Current benchmarks mainly measure reasoning and factual recall, not true multilingual proficiency.

Round-trip translation requires no human references and aligns well with real-world multilingual tasks.

Abstract

Multilingual benchmarks guide the development of frontier models. Yet multilingual evaluations reported by frontier models are structured similar to popular reasoning and knowledge benchmarks, but across many languages. We show such benchmarks, and consequently multilingual evaluations, measure mathematical reasoning and factual recall, not multilingual proficiency. For example, thinking variants dramatically outperform instruct variants on these benchmarks, yet often perform worse on real-world multilingual tasks, such as LMArena. We propose a simple alternative: evaluate multilingual capability via round-trip translation. Given text in a source language, translate it to a target language and back; semantic gaps between the original and result expose failures in multilingual generation capabilities. Round-trip translation correlates almost perfectly (\r{ho} = 0.94) with user ratings on LMArena with our benchmark, requires no human reference translations, and does not require a more capable multilingual judge than tested models. Lastly, we introduce Lost in Translation (LiT), a challenging round-trip translation benchmark spanning widely spoken languages worldwide, for realistic evaluation of multilingual frontier models.