The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failure

TL;DR

This paper identifies an implicit translation failure as a key reason for poor multilingual generation quality in large language models, especially for low-resource languages, by analyzing a two-stage task-solving and translation pipeline.

Contribution

It introduces the translation barrier hypothesis, formalizes it, and quantifies its impact across 108 language pairs, revealing it as a major bottleneck in multilingual LLM generation.

Findings

Translation barrier explains most errors in low-resource languages.

Model first solves task in a language-agnostic way, then translates.

Translation failure is a critical bottleneck for multilingual LLMs.

Abstract

Multilingual generation with large language models (LLMs) is often of poor quality for mid- to low-resource languages, but the causes for this are not well-understood. We first demonstrate the existence of an implicit task-solving-->translation pipeline for generation, whereby the model first solves the required task in a largely target-language-agnostic manner, and subsequently translates answer concepts into the intended target language. We hypothesize that the failure of the translation stage, despite task-solving success, is an important culprit for the observed low quality of final outputs, and formalize this as the translation barrier hypothesis. We quantify the extent to which either stage in the pipeline is responsible for final failure for a word translation task across 108 language pairs, and find that the translation barrier explains a dominant portion of error for a majority of language pairs, and is especially severe for low-resource target languages. Our results highlight an important bottleneck for end-to-end multilingual generation, relevant for future work seeking to improve multilinguality in LLMs.