Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMs

TL;DR

This paper introduces new metrics to evaluate the naturalness of multilingual LLM outputs, revealing English bias, and proposes an alignment method to improve naturalness in non-English languages without losing performance.

Contribution

The paper develops automatic corpus-level metrics for assessing multilingual LLM naturalness and proposes an alignment method to reduce English bias in generated outputs.

Findings

Multilingual LLMs tend to produce English-influenced patterns.

New metrics effectively measure lexical and syntactic naturalness.

Alignment method improves naturalness without harming benchmark performance.

Abstract

Current Large Language Models (LLMs) are predominantly designed with English as the primary language, and even the few that are multilingual tend to exhibit strong English-centric biases. Much like speakers who might produce awkward expressions when learning a second language, LLMs often generate unnatural outputs in non-English languages, reflecting English-centric patterns in both vocabulary and grammar. Despite the importance of this issue, the naturalness of multilingual LLM outputs has received limited attention. In this paper, we address this gap by introducing novel automatic corpus-level metrics to assess the lexical and syntactic naturalness of LLM outputs in a multilingual context. Using our new metrics, we evaluate state-of-the-art LLMs on a curated benchmark in French and Chinese, revealing a tendency towards English-influenced patterns. To mitigate this issue, we also propose a simple and effective alignment method to improve the naturalness of an LLM in a target language and domain, achieving consistent improvements in naturalness without compromising the performance on general-purpose benchmarks. Our work highlights the importance of developing multilingual metrics, resources and methods for the new wave of multilingual LLMs.