Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editing

TL;DR

This paper explores using large language models as automatic post-editors for translation, achieving state-of-the-art accuracy in pronoun resolution and demonstrating the benefits of manual feedback in document-level translation.

Contribution

It introduces a novel approach of adapting LLMs as post-editors with Low-Rank-Adapter fine-tuning, improving translation quality and generalization, especially at document level.

Findings

Achieved 89% accuracy on ContraPro pronoun resolution test.

Fine-tuning as post-editors outperforms direct translation fine-tuning.

Manual corrections significantly reduce subsequent editing effort.

Abstract

Large Language Models (LLM's) have demonstrated considerable success in various Natural Language Processing tasks, but they have yet to attain state-of-the-art performance in Neural Machine Translation (NMT). Nevertheless, their significant performance in tasks demanding a broad understanding and contextual processing shows their potential for translation. To exploit these abilities, we investigate using LLM's for MT and explore recent parameter-efficient fine-tuning techniques. Surprisingly, our initial experiments find that fine-tuning for translation purposes even led to performance degradation. To overcome this, we propose an alternative approach: adapting LLM's as Automatic Post-Editors (APE) rather than direct translators. Building on the LLM's exceptional ability to process and generate lengthy sequences, we also propose extending our approach to document-level translation. We show that leveraging Low-Rank-Adapter fine-tuning for APE can yield significant improvements across both sentence and document-level metrics while generalizing to out-of-domain data. Most notably, we achieve a state-of-the-art accuracy rate of 89\% on the ContraPro test set, which specifically assesses the model's ability to resolve pronoun ambiguities when translating from English to German. Lastly, we investigate a practical scenario involving manual post-editing for document-level translation, where reference context is made available. Here, we demonstrate that leveraging human corrections can significantly reduce the number of edits required for subsequent translations (Interactive Demo for integrating manual feedback can be found here: https://huggingface.co/spaces/skoneru/contextual_refinement_ende).