A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models

TL;DR

This paper introduces a novel two-stage fine-tuning method for moderate-sized LLMs that significantly improves translation quality, surpassing larger models and traditional methods without relying on extensive parallel data.

Contribution

The study presents ALMA, a new fine-tuning approach that enhances translation performance of 7B and 13B LLMs using monolingual and minimal high-quality parallel data, establishing a new training paradigm.

Findings

Achieved over 12 BLEU and 12 COMET improvements across 10 translation directions.

Outperformed larger models like NLLB-54B and GPT-3.5 in translation tasks.

Demonstrated effectiveness of the two-stage fine-tuning strategy for machine translation.

Abstract

Generative Large Language Models (LLMs) have achieved remarkable advancements in various NLP tasks. However, these advances have not been reflected in the translation task, especially those with moderate model sizes (i.e., 7B or 13B parameters), which still lag behind conventional supervised encoder-decoder translation models. Previous studies have attempted to improve the translation capabilities of these moderate LLMs, but their gains have been limited. In this study, we propose a novel fine-tuning approach for LLMs that is specifically designed for the translation task, eliminating the need for the abundant parallel data that traditional translation models usually depend on. Our approach consists of two fine-tuning stages: initial fine-tuning on monolingual data followed by subsequent fine-tuning on a small set of high-quality parallel data. We introduce the LLM developed through this strategy as Advanced Language Model-based trAnslator (ALMA). Based on LLaMA-2 as our underlying model, our results show that the model can achieve an average improvement of more than 12 BLEU and 12 COMET over its zero-shot performance across 10 translation directions from the WMT'21 (2 directions) and WMT'22 (8 directions) test datasets. The performance is significantly better than all prior work and even superior to the NLLB-54B model and GPT-3.5-text-davinci-003, with only 7B or 13B parameters. This method establishes the foundation for a novel training paradigm in machine translation.