Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translation

TL;DR

This paper investigates how speed-optimization techniques in neural machine translation models can unintentionally amplify gender bias, especially in translating gendered nouns, despite minimal impact on overall BLEU scores.

Contribution

It introduces SimpleGEN, a new gender bias test set, and analyzes the impact of common speed-up methods on gender bias amplification in NMT models.

Findings

Gender bias in translation increases with speed optimizations.

Overall BLEU scores remain relatively stable despite bias amplification.

Speed-optimized models show faster degradation in gendered noun translation quality.

Abstract

Is bias amplified when neural machine translation (NMT) models are optimized for speed and evaluated on generic test sets using BLEU? We investigate architectures and techniques commonly used to speed up decoding in Transformer-based models, such as greedy search, quantization, average attention networks (AANs) and shallow decoder models and show their effect on gendered noun translation. We construct a new gender bias test set, SimpleGEN, based on gendered noun phrases in which there is a single, unambiguous, correct answer. While we find minimal overall BLEU degradation as we apply speed optimizations, we observe that gendered noun translation performance degrades at a much faster rate.