Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss
Ronald Skorobogat, Ameya Prabhu, Matthias Bethge

TL;DR
This paper critiques current multilingual benchmarks for not accurately measuring multilingual proficiency and proposes round-trip translation as a more effective evaluation method, introducing the LiT benchmark.
Contribution
It introduces round-trip translation as a novel evaluation method for multilingual models and presents the LiT benchmark for realistic multilingual assessment.
Findings
Round-trip translation correlates highly (0.94) with user ratings on LMArena.
Current benchmarks mainly measure reasoning and factual recall, not true multilingual proficiency.
Round-trip translation requires no human references and aligns well with real-world multilingual tasks.
Abstract
Multilingual benchmarks guide the development of frontier models. Yet multilingual evaluations reported by frontier models are structured similar to popular reasoning and knowledge benchmarks, but across many languages. We show such benchmarks, and consequently multilingual evaluations, measure mathematical reasoning and factual recall, not multilingual proficiency. For example, thinking variants dramatically outperform instruct variants on these benchmarks, yet often perform worse on real-world multilingual tasks, such as LMArena. We propose a simple alternative: evaluate multilingual capability via round-trip translation. Given text in a source language, translate it to a target language and back; semantic gaps between the original and result expose failures in multilingual generation capabilities. Round-trip translation correlates almost perfectly (\r{ho} = 0.94) with user ratings on…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
