Lesan -- Machine Translation for Low Resource Languages

TL;DR

Lesan is a new machine translation system designed for low-resource languages, leveraging innovative data collection, OCR, and sequence-to-sequence models to outperform existing systems and improve web access for underserved language speakers.

Contribution

Lesan introduces a pipeline combining online/offline data sources, OCR, and automatic alignment with Transformer-based models for low-resource language translation.

Findings

Lesan outperforms Google Translate and Microsoft Translator in human evaluations.

Supports translation for Tigrinya, Amharic, and English.

Has served over 10 million translations to date.

Abstract

Millions of people around the world can not access content on the Web because most of the content is not readily available in their language. Machine translation (MT) systems have the potential to change this for many languages. Current MT systems provide very accurate results for high resource language pairs, e.g., German and English. However, for many low resource languages, MT is still under active research. The key challenge is lack of datasets to build these systems. We present Lesan, an MT system for low resource languages. Our pipeline solves the key bottleneck to low resource MT by leveraging online and offline sources, a custom OCR system for Ethiopic and an automatic alignment module. The final step in the pipeline is a sequence to sequence model that takes parallel corpus as input and gives us a translation model. Lesan's translation model is based on the Transformer architecture. After constructing a base model, back translation, is used to leverage monolingual corpora. Currently Lesan supports translation to and from Tigrinya, Amharic and English. We perform extensive human evaluation and show that Lesan outperforms state-of-the-art systems such as Google Translate and Microsoft Translator across all six pairs. Lesan is freely available and has served more than 10 million translations so far. At the moment, there are only 217 Tigrinya and 15,009 Amharic Wikipedia articles. We believe that Lesan will contribute towards democratizing access to the Web through MT for millions of people.