Do LLMs Truly Benefit from Longer Context in Automatic Post-Editing?

TL;DR

This paper systematically evaluates large language models for automatic post-editing, revealing that while proprietary models perform well with simple prompts, they do not effectively utilize document context and are costly, highlighting the need for more efficient solutions.

Contribution

It provides a comprehensive comparison of proprietary and open-weight LLMs for document-level APE, analyzing their performance, robustness, and limitations in context utilization.

Findings

Proprietary LLMs achieve near human-level APE quality with simple prompts.

Open-weight models are less robust but better at exploiting document context.

Current automatic metrics do not reliably reflect qualitative improvements.

Abstract

Automatic post-editing (APE) aims to refine machine translations by correcting residual errors. Although recent large language models (LLMs) demonstrate strong translation capabilities, their effectiveness for APE--especially under document-level context--remains insufficiently understood. We present a systematic comparison of proprietary and open-weight LLMs under a naive document-level prompting setup, analyzing APE quality, contextual behavior, robustness, and efficiency. Our results show that proprietary LLMs achieve near human-level APE quality even with simple one-shot prompting, regardless of whether document context is provided. While these models exhibit higher robustness to data poisoning attacks than open-weight counterparts, this robustness also reveals a limitation: they largely fail to exploit document-level context for contextual error correction. Furthermore, standard automatic metrics do not reliably reflect these qualitative improvements, highlighting the continued necessity of human evaluation. Despite their strong performance, the substantial cost and latency overheads of proprietary LLMs render them impractical for real-world APE deployment. Overall, our findings elucidate both the promise and current limitations of LLM-based document-aware APE, and point toward the need for more efficient long-context modeling approaches for translation refinement.