Textual Data Bias Detection and Mitigation -- An Extensible Pipeline with Experimental Evaluation

TL;DR

This paper presents an extensible pipeline for detecting and mitigating textual data bias, including representation bias and stereotypes, to improve fairness in large language models, with comprehensive evaluation on multiple bias types and models.

Contribution

It introduces a novel, configurable pipeline combining LLM-generated labels, sociolinguistic filtering, and counterfactual augmentation for bias mitigation in textual datasets.

Findings

Successfully reduces representation bias and stereotypes in datasets.

Debiased data does not always lead to improved model bias benchmarks.

Highlights gaps in current bias evaluation methodologies.

Abstract

Textual data used to train large language models (LLMs) exhibits multifaceted bias manifestations encompassing harmful language and skewed demographic distributions. Regulations such as the European AI Act require identifying and mitigating biases against protected groups in data, with the ultimate goal of preventing unfair model outputs. However, practical guidance and operationalization are lacking. We propose a comprehensive data bias detection and mitigation pipeline comprising four components that address two data bias types, namely representation bias and (explicit) stereotypes for a configurable sensitive attribute. First, we leverage LLM-generated word lists created based on quality criteria to detect relevant group labels. Second, representation bias is quantified using the Demographic Representation Score. Third, we detect and mitigate stereotypes using sociolinguistically informed filtering. Finally, we compensate representation bias through Grammar- and Context-Aware Counterfactual Data Augmentation. We conduct a two-fold evaluation using the examples of gender, religion and age. First, the effectiveness of each individual component on data debiasing is evaluated through human validation and baseline comparison. The findings demonstrate that we successfully reduce representation bias and (explicit) stereotypes in a text dataset. Second, the effect of data debiasing on model bias reduction is evaluated by bias benchmarking of several models (0.6B-8B parameters), fine-tuned on the debiased text dataset. This evaluation reveals that LLMs fine-tuned on debiased data do not consistently show improved performance on bias benchmarks, exposing critical gaps in current evaluation methodologies and highlighting the need for targeted data manipulation to address manifested model bias.