GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations

TL;DR

This paper introduces GECOBench, a dataset and evaluation framework to quantify gender bias in explanations generated by XAI methods for NLP models, revealing how fine-tuning affects bias mitigation.

Contribution

It presents a novel gender-controlled dataset and benchmark for evaluating explanation bias in language models, and analyzes the impact of fine-tuning on bias reduction in feature attributions.

Findings

Fine-tuning reduces explanation bias in XAI methods.

Explanation performance improves with more fine-tuned layers.

GECOBench enables objective evaluation of bias in explanations.

Abstract

Large pre-trained language models have become a crucial backbone for many downstream tasks in natural language processing (NLP), and while they are trained on a plethora of data containing a variety of biases, such as gender biases, it has been shown that they can also inherit such biases in their weights, potentially affecting their prediction behavior. However, it is unclear to what extent these biases also affect feature attributions generated by applying "explainable artificial intelligence" (XAI) techniques, possibly in unfavorable ways. To systematically study this question, we create a gender-controlled text dataset, GECO, in which the alteration of grammatical gender forms induces class-specific words and provides ground truth feature attributions for gender classification tasks. This enables an objective evaluation of the correctness of XAI methods. We apply this dataset to the pre-trained BERT model, which we fine-tune to different degrees, to quantitatively measure how pre-training induces undesirable bias in feature attributions and to what extent fine-tuning can mitigate such explanation bias. To this extent, we provide GECOBench, a rigorous quantitative evaluation framework for benchmarking popular XAI methods. We show a clear dependency between explanation performance and the number of fine-tuned layers, where XAI methods are observed to benefit particularly from fine-tuning or complete retraining of embedding layers.