How Important is `Perfect' English for Machine Translation Prompts?

TL;DR

This paper investigates how various types of prompt errors influence large language models' performance in machine translation and evaluation, revealing that prompt quality significantly impacts results and models can tolerate substantial noise.

Contribution

It systematically analyzes the effects of human-like and synthetic prompt errors on LLM translation tasks, providing insights into prompt robustness and model behavior under noisy conditions.

Findings

Prompt quality strongly influences translation performance.

Different noise types affect models differently, with character-level noise being most damaging.

Models can still perform translation despite high levels of random noise.

Abstract

Large language models (LLMs) have achieved top results in recent machine translation evaluations, but they are also known to be sensitive to errors and perturbations in their prompts. We systematically evaluate how both humanly plausible and synthetic errors in user prompts affect LLMs' performance on two related tasks: Machine translation and machine translation evaluation. We provide both a quantitative analysis and qualitative insights into how the models respond to increasing noise in the user prompt. The prompt quality strongly affects the translation performance: With many errors, even a good prompt can underperform a minimal or poor prompt without errors. However, different noise types impact translation quality differently, with character-level and combined noisers degrading performance more than phrasal perturbations. Qualitative analysis reveals that lower prompt quality largely leads to poorer instruction following, rather than directly affecting translation quality itself. Further, LLMs can still translate in scenarios with overwhelming random noise that would make the prompt illegible to humans.