Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventions

TL;DR

This paper introduces a simple, practical few-shot data intervention method to significantly reduce gender bias in pre-trained language models, outperforming existing techniques with minimal impact on language modeling performance.

Contribution

The paper proposes a novel few-shot data intervention approach for debiasing language models, requiring only 10 examples, which is more efficient and effective than current methods.

Findings

Debiasing with only 10 examples significantly reduces gender bias.

The method outperforms state-of-the-art baselines.

Minimal loss in language modeling ability.

Abstract

Societal biases present in pre-trained large language models are a critical issue as these models have been shown to propagate biases in countless downstream applications, rendering them unfair towards specific groups of people. Since large-scale retraining of these models from scratch is both time and compute-expensive, a variety of approaches have been previously proposed that de-bias a pre-trained model. While the majority of current state-of-the-art debiasing methods focus on changes to the training regime, in this paper, we propose data intervention strategies as a powerful yet simple technique to reduce gender bias in pre-trained models. Specifically, we empirically show that by fine-tuning a pre-trained model on only 10 de-biased (intervened) training examples, the tendency to favor any gender is significantly reduced. Since our proposed method only needs a few training examples, our few-shot debiasing approach is highly feasible and practical. Through extensive experimentation, we show that our debiasing technique performs better than competitive state-of-the-art baselines with minimal loss in language modeling ability.