Language verY Rare for All

TL;DR

This paper introduces LYRA, a novel approach combining open LLM fine-tuning, retrieval-augmented generation, and transfer learning to improve translation of rare languages like Monégasque using only a single GPU.

Contribution

LYRA is a new method that enhances rare language translation by integrating fine-tuning, retrieval, and transfer learning, enabling effective single-GPU training.

Findings

LYRA outperforms existing models in rare language translation.

LYRA matches state-of-the-art encoder-decoder models.

The approach is effective for low-resource language pairs.

Abstract

In the quest to overcome language barriers, encoder-decoder models like NLLB have expanded machine translation to rare languages, with some models (e.g., NLLB 1.3B) even trainable on a single GPU. While general-purpose LLMs perform well in translation, open LLMs prove highly competitive when fine-tuned for specific tasks involving unknown corpora. We introduce LYRA (Language verY Rare for All), a novel approach that combines open LLM fine-tuning, retrieval-augmented generation (RAG), and transfer learning from related high-resource languages. This study is exclusively focused on single-GPU training to facilitate ease of adoption. Our study focuses on two-way translation between French and Mon\'egasque, a rare language unsupported by existing translation tools due to limited corpus availability. Our results demonstrate LYRA's effectiveness, frequently surpassing and consistently matching state-of-the-art encoder-decoder models in rare language translation.