fairBERTs: Erasing Sensitive Information Through Semantic and Fairness-aware Perturbations

TL;DR

fairBERTs introduces a framework that uses semantic and fairness-aware perturbations via GANs to erase sensitive information in BERT models, reducing bias while preserving utility.

Contribution

This work proposes a novel method for fine-tuning BERT models to mitigate biases by erasing sensitive info through adversarial perturbations, enhancing fairness in NLP applications.

Findings

fairBERTs effectively reduces bias in real-world tasks

The method maintains high model utility after bias mitigation

Adversarial components can be transferred to other BERT models

Abstract

Pre-trained language models (PLMs) have revolutionized both the natural language processing research and applications. However, stereotypical biases (e.g., gender and racial discrimination) encoded in PLMs have raised negative ethical implications for PLMs, which critically limits their broader applications. To address the aforementioned unfairness issues, we present fairBERTs, a general framework for learning fair fine-tuned BERT series models by erasing the protected sensitive information via semantic and fairness-aware perturbations generated by a generative adversarial network. Through extensive qualitative and quantitative experiments on two real-world tasks, we demonstrate the great superiority of fairBERTs in mitigating unfairness while maintaining the model utility. We also verify the feasibility of transferring adversarial components in fairBERTs to other conventionally trained BERT-like models for yielding fairness improvements. Our findings may shed light on further research on building fairer fine-tuned PLMs.