FineDeb: A Debiasing Framework for Language Models

TL;DR

FineDeb is a two-phase debiasing framework for language models that effectively reduces demographic biases while maintaining language modeling performance, outperforming existing methods.

Contribution

It introduces a novel two-step debiasing approach combining contextual embedding debiasing with fine-tuning, applicable to multiple demographic classes.

Findings

Stronger debiasing compared to other methods

Effective across multiple demographic groups

Maintains language modeling quality

Abstract

As language models are increasingly included in human-facing machine learning tools, bias against demographic subgroups has gained attention. We propose FineDeb, a two-phase debiasing framework for language models that starts with contextual debiasing of embeddings learned by pretrained language models. The model is then fine-tuned on a language modeling objective. Our results show that FineDeb offers stronger debiasing in comparison to other methods which often result in models as biased as the original language model. Our framework is generalizable for demographics with multiple classes, and we demonstrate its effectiveness through extensive experiments and comparisons with state of the art techniques. We release our code and data on GitHub.