Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning
Changtong Zan, Liang Ding, Li Shen, Yibing Zhen, Weifeng Liu, Dacheng, Tao

TL;DR
This paper presents a two-stage fine-tuning method for large language models to improve translation accuracy and instruction-following, especially for low-resource languages, by reducing off-target translations.
Contribution
It introduces a novel two-stage fine-tuning approach with an unlikelihood loss to enhance translation direction accuracy in LLMs.
Findings
Reduces off-target translation ratio by 53.3%
Improves translation quality by 5.7 BLEU and 16.4 BLEURT
Preserves general task performance on AlpacaEval
Abstract
Translation-tailored Large language models (LLMs) exhibit remarkable translation capabilities, even competing with supervised-trained commercial translation systems. However, off-target translation remains an unsolved problem, especially for low-resource languages, hindering us from developing accurate LLMs-based translation models. To mitigate the off-target translation problem and enhance the performance of LLMs on translation, recent works have either designed advanced prompting strategies to highlight the functionality of translation instructions or exploited the in-context learning ability of LLMs by feeding few-shot demonstrations. However, these methods essentially do not improve LLM's ability to follow translation instructions, especially the language direction information. In this work, we design a two-stage fine-tuning algorithm to improve the instruction-following ability…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques
MethodsLLaMA
