Downstream bias mitigation is all you need

TL;DR

This paper investigates bias in large language models, revealing that biases are more effectively mitigated by addressing domain-specific data during fine-tuning rather than through interventions on pre-trained models.

Contribution

It demonstrates that bias mitigation is more effective when applied to fine-tuning datasets rather than pre-trained models, emphasizing the importance of domain-specific data.

Findings

Biases in LLMs are minimally affected by pre-training interventions.

Bias mitigation at the fine-tuning stage has a larger impact.

Small changes in fine-tuning data co-occurrence rates significantly affect bias.

Abstract

The advent of transformer-based architectures and large language models (LLMs) have significantly advanced the performance of natural language processing (NLP) models. Since these LLMs are trained on huge corpuses of data from the web and other sources, there has been a major concern about harmful prejudices that may potentially be transferred from the data. In many applications, these pre-trained LLMs are fine-tuned on task specific datasets, which can further contribute to biases. This paper studies the extent of biases absorbed by LLMs during pre-training as well as task-specific behaviour after fine-tuning. We found that controlled interventions on pre-trained LLMs, prior to fine-tuning, have minimal effect on lowering biases in classifiers. However, the biases present in domain-specific datasets play a much bigger role, and hence mitigating them at this stage has a bigger impact. While pre-training does matter, but after the model has been pre-trained, even slight changes to co-occurrence rates in the fine-tuning dataset has a significant effect on the bias of the model.