Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors

TL;DR

This study evaluates how quality estimation and backtranslation techniques assist physicians in making informed decisions about using machine translation outputs in high-stakes medical settings, highlighting their complementary benefits.

Contribution

It provides the first in vivo human study comparing quality estimation and backtranslation for clinical decision-making in machine translation, demonstrating their respective strengths.

Findings

Quality estimation improves appropriate reliance on MT outputs.

Backtranslation helps detect more clinically harmful errors.

Combined use enhances decision-making accuracy.

Abstract

A major challenge in the practical use of Machine Translation (MT) is that users lack guidance to make informed decisions about when to rely on outputs. Progress in quality estimation research provides techniques to automatically assess MT quality, but these techniques have primarily been evaluated in vitro by comparison against human judgments outside of a specific context of use. This paper evaluates quality estimation feedback in vivo with a human study simulating decision-making in high-stakes medical settings. Using Emergency Department discharge instructions, we study how interventions based on quality estimation versus backtranslation assist physicians in deciding whether to show MT outputs to a patient. We find that quality estimation improves appropriate reliance on MT, but backtranslation helps physicians detect more clinically harmful errors that QE alone often misses.