Uncertainty Quantification for Evaluating Machine Translation Bias

TL;DR

This paper explores how uncertainty quantification in machine translation can be used to detect gender bias, especially in ambiguous cases, revealing that high accuracy does not imply proper bias mitigation.

Contribution

It introduces a novel approach using semantic uncertainty to measure gender bias in MT, applicable to ambiguous and unambiguous sentences, and analyzes the impact of debiasing.

Findings

Uncertainty can reveal bias in ambiguous translations.

High accuracy does not guarantee unbiased translations.

Debiasing affects ambiguous and unambiguous cases differently.

Abstract

The predictive uncertainty of machine translation (MT) models is typically used as a quality estimation proxy. In this work, we posit that apart from confidently translating when a single correct translation exists, models should also maintain uncertainty when the input is ambiguous. We use uncertainty to measure gender bias in MT systems. When the source sentence includes a lexeme whose gender is not overtly marked, but whose target-language equivalent requires gender specification, the model must infer the appropriate gender from the context and can be susceptible to biases. Prior work measured bias via gender accuracy, however it cannot be applied to ambiguous cases. Using semantic uncertainty, we are able to assess bias when translating both ambiguous and unambiguous source sentences, and find that high translation accuracy does not correlate with exhibiting uncertainty appropriately, and that debiasing affects the two cases differently.