Sustainable Modular Debiasing of Language Models

TL;DR

This paper introduces ADELE, a modular debiasing method for language models that uses dedicated adapters to effectively reduce bias while preserving language knowledge and enabling multilingual transfer.

Contribution

The paper presents ADELE, a novel modular debiasing approach that updates only adapter modules in PLMs, improving efficiency and flexibility over traditional methods.

Findings

ADELE effectively reduces gender bias in BERT across multiple bias measures.

The modular approach retains fairness after downstream task training.

ADELE successfully transfers to six languages using multilingual BERT.

Abstract

Unfair stereotypical biases (e.g., gender, racial, or religious biases) encoded in modern pretrained language models (PLMs) have negative ethical implications for widespread adoption of state-of-the-art language technology. To remedy for this, a wide range of debiasing techniques have recently been introduced to remove such stereotypical biases from PLMs. Existing debiasing methods, however, directly modify all of the PLMs parameters, which -- besides being computationally expensive -- comes with the inherent risk of (catastrophic) forgetting of useful language knowledge acquired in pretraining. In this work, we propose a more sustainable modular debiasing approach based on dedicated debiasing adapters, dubbed ADELE. Concretely, we (1) inject adapter modules into the original PLM layers and (2) update only the adapters (i.e., we keep the original PLM parameters frozen) via language modeling training on a counterfactually augmented corpus. We showcase ADELE, in gender debiasing of BERT: our extensive evaluation, encompassing three intrinsic and two extrinsic bias measures, renders ADELE, very effective in bias mitigation. We further show that -- due to its modular nature -- ADELE, coupled with task adapters, retains fairness even after large-scale downstream training. Finally, by means of multilingual BERT, we successfully transfer ADELE, to six target languages.