Towards Bridging Language Gaps in OSS with LLM-Driven Documentation Translation

TL;DR

This paper evaluates the potential of large language models like ChatGPT 4 and Anthropic Claude to translate open source technical documentation, highlighting their strengths in semantic accuracy and challenges in structural preservation.

Contribution

It provides an empirical assessment of LLMs for translating OSS documentation, revealing their capabilities and limitations in real-world scenarios.

Findings

LLMs can produce accurate semantic translations of technical content.

Translation activity is mainly community-driven and concentrated in larger repositories.

LLMs struggle to preserve document structure and technical formatting.

Abstract

While open source communities attract diverse contributors across the globe, only a few open source software repositories provide essential documentation, such as ReadMe or CONTRIBUTING files, in languages other than English. Recently, large language models (LLMs) have demonstrated remarkable capabilities in a variety of software engineering tasks. We have also seen advances in the use of LLMs for translations in other domains and contexts. Despite this progress, little is known regarding the capabilities of LLMs in translating open-source technical documentation, which is often a mixture of natural language, code, URLs, and markdown formatting. To better understand the need and potential for LLMs to support translation of technical documentation in open source, we conducted an empirical evaluation of translation activity and translation capabilities of two powerful large language models (OpenAI ChatGPT 4 and Anthropic Claude). We found that translation activity is often community-driven and most frequent in larger repositories. A comparison of LLM performance as translators and evaluators of technical documentation suggests LLMs can provide accurate semantic translations but may struggle preserving structure and technical content. These findings highlight both the promise and the challenges of LLM-assisted documentation internationalization and provide a foundation towards automated LLM-driven support for creating and maintaining open source documentation.