CycleDistill: Bootstrapping Machine Translation using LLMs with Cyclical Distillation

TL;DR

CycleDistill is a novel bootstrapping method that uses LLMs and cyclical distillation to improve machine translation for low-resource languages without requiring extensive parallel data.

Contribution

It introduces a cyclical distillation approach that leverages LLMs and minimal monolingual data to enhance translation quality in low-resource settings.

Findings

Achieves 20-30 chrF points improvement over baseline

Effective with only 1-4 few-shot examples

Softmax activation during distillation yields mild gains

Abstract

Large language models (LLMs), despite their ability to perform few-shot machine translation (MT), often lag behind dedicated MT systems trained on parallel corpora, which are crucial for high quality machine translation (MT). However, parallel corpora are often scarce or non-existent for low-resource languages. In this paper, we propose CycleDistill, a bootstrapping approach leveraging LLMs and few-shot translation to obtain high-quality MT systems. CycleDistill involves iteratively generating synthetic parallel corpora from monolingual corpora via zero- or few-shot MT, which is then used to fine-tune the model that was used for generating said data for MT. CycleDistill does not need parallel corpora beyond 1 to 4 few-shot examples, and in our experiments focusing on three Indian languages, by relying solely on monolingual corpora, it can achieve high-quality machine translation, improving upon a few-shot baseline model by over 20-30 chrF points on average in the first iteration. We also study the effect of leveraging softmax activations during the distillation process and observe mild improvements in translation quality.