Should We be Pedantic About Reasoning Errors in Machine Translation?

TL;DR

This paper investigates reasoning errors in machine translation across multiple languages, showing that while such errors can be identified with high precision, correcting them has limited impact on translation quality.

Contribution

It introduces an automated protocol for detecting reasoning errors in MT and evaluates the effect of various interventions on translation quality.

Findings

High-precision detection of reasoning errors in Urdu

Lower detection precision in Spanish

Interventions often do not significantly improve translation quality

Abstract

Across multiple language pairings (English $\to$ \{Spanish, French, German, Mandarin, Japanese, Urdu, Cantonese\}), we find reasoning errors in translation. To quantify how often these reasoning errors occur, we leverage an automated annotation protocol for reasoning evaluation wherein the goal is to detect if a reasoning step is any of three error categories: (1) source sentence-misaligned, (2) model hypothesis-misaligned, or (3) reasoning trace-misaligned. We probe the reasoning model with perturbed traces correcting for these identified reasoning errors using an array of weak-to-strong interventions: hedging, removal, re-reasoning after removal, hindsight, and oracle interventions. Experimenting with interventions on the reasoning traces suggests that small corrections to the reasoning have little impact on translation quality, but stronger interventions yield the highest resolution rates, despite translation quality gains being mixed. We find ultimately that reasoning errors in MT can be identified with high precision in Urdu but lower precision in Spanish, but that removing these reasoning errors does not resolve the initial errors significantly, suggesting limited reasoning faithfulness for machine translation.