Science Across Languages: Assessing LLM Multilingual Translation of Scientific Papers

TL;DR

This paper presents an automated LLM-based method for translating scientific articles into multiple languages while preserving formatting, evaluating accuracy through a novel QA benchmark and user studies, demonstrating high fidelity and adaptability.

Contribution

It introduces a practical, automated approach for multilingual scientific translation using LLMs, including a novel QA benchmark and techniques to align translations with domain preferences.

Findings

Average translation accuracy of 95.9% on scientific content

Authors found translations accurately captured original information

In-context learning can mitigate overtranslation issues

Abstract

Scientific research is inherently global. However, the vast majority of academic journals are published exclusively in English, creating barriers for non-native-English-speaking researchers. In this study, we leverage large language models (LLMs) to translate published scientific articles while preserving their native JATS XML formatting, thereby developing a practical, automated approach for implementation by academic journals. Using our approach, we translate articles across multiple scientific disciplines into 28 languages. To evaluate translation accuracy, we introduce a novel question-and-answer (QA) benchmarking method, in which an LLM generates comprehension-based questions from the original text and then answers them based on the translated text. Our benchmark results show an average performance of 95.9%, showing that the key scientific details are accurately conveyed. In a user study, we translate the scientific papers of 15 researchers into their native languages, finding that the authors consistently found the translations to accurately capture the original information in their articles. Interestingly, a third of the authors found many technical terms "overtranslated," expressing a preference to keep terminology more familiar in English untranslated. Finally, we demonstrate how in-context learning techniques can be used to align translations with domain-specific preferences such as mitigating overtranslation, highlighting the adaptability and utility of LLM-driven scientific translation. The code and translated articles are available at https://hankleid.github.io/ProjectMundo.