Applying Intrinsic Debiasing on Downstream Tasks: Challenges and Considerations for Machine Translation

TL;DR

This paper investigates how intrinsic gender bias debiasing methods impact neural machine translation systems, revealing challenges and considerations that affect downstream performance and debiasing effectiveness.

Contribution

It systematically evaluates the effects of intrinsic debiasing techniques on machine translation, highlighting key challenges and mismatches with end-goal applications.

Findings

Debiasing choices significantly influence downstream translation quality.

Mismatch between word and sub-word debiasing affects bias reduction.

Debiasing impacts vary across target languages.

Abstract

Most works on gender bias focus on intrinsic bias -- removing traces of information about a protected group from the model's internal representation. However, these works are often disconnected from the impact of such debiasing on downstream applications, which is the main motivation for debiasing in the first place. In this work, we systematically test how methods for intrinsic debiasing affect neural machine translation models, by measuring the extrinsic bias of such systems under different design choices. We highlight three challenges and mismatches between the debiasing techniques and their end-goal usage, including the choice of embeddings to debias, the mismatch between words and sub-word tokens debiasing, and the effect on different target languages. We find that these considerations have a significant impact on downstream performance and the success of debiasing.