Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models

TL;DR

This study examines bias in large language models by evaluating various proxy measures, revealing that task formulation significantly influences bias detection, especially in personalized and socio-demographic contexts.

Contribution

It provides a comparative analysis of bias measures in LLMs and highlights how different task formulations impact bias detection and assessment.

Findings

Pre-prompted personae yield negligible bias differences.

Answer grading tasks reveal more significant bias.

Bias is pronounced in salary negotiation advice.

Abstract

Modern language models are trained on large amounts of data. These data inevitably include controversial and stereotypical content, which contains all sorts of biases related to gender, origin, age, etc. As a result, the models express biased points of view or produce different results based on the assigned personality or the personality of the user. In this paper, we investigate various proxy measures of bias in large language models (LLMs). We find that evaluating models with pre-prompted personae on a multi-subject benchmark (MMLU) leads to negligible and mostly random differences in scores. However, if we reformulate the task and ask a model to grade the user's answer, this shows more significant signs of bias. Finally, if we ask the model for salary negotiation advice, we see pronounced bias in the answers. With the recent trend for LLM assistant memory and personalization, these problems open up from a different angle: modern LLM users do not need to pre-prompt the description of their persona since the model already knows their socio-demographics.