Open Machine Translation for Esperanto

TL;DR

This paper evaluates open-source machine translation systems for Esperanto, comparing various models and providing insights into their performance through automatic and human assessments.

Contribution

It is the first comprehensive evaluation of open-source MT systems for Esperanto, including rule-based, encoder-decoder, and LLM approaches, with publicly released code and models.

Findings

NLLB models outperform others across all language pairs.

Compact models and fine-tuned LLMs perform closely to NLLB.

Human evaluation favors NLLB translations in about half of the cases.

Abstract

Esperanto is a widespread constructed language, known for its regular grammar and productive word formation. Besides having substantial resources available thanks to its online community, it remains relatively underexplored in the context of modern machine translation (MT) approaches. In this work, we present the first comprehensive evaluation of open-source MT systems for Esperanto, comparing rule-based systems, encoder-decoder models, and LLMs across model sizes. We evaluate translation quality across six language directions involving English, Spanish, Catalan, and Esperanto using multiple automatic metrics as well as human evaluation. Our results show that the NLLB family achieves the best performance in all language pairs, followed closely by our trained compact models and a fine-tuned general-purpose LLM. Human evaluation confirms this trend, with NLLB translations preferred in approximately half of the comparisons, although noticeable errors remain. In line with Esperanto's tradition of openness and international collaboration, we release our code and best-performing models publicly.