Prompt Fairness: Sub-group Disparities in LLMs

TL;DR

This paper investigates prompt fairness in LLMs, quantifies subgroup disparities using information-theoretic metrics, and proposes mitigation strategies that improve response consistency and fairness across demographic groups.

Contribution

It introduces a novel framework for measuring prompt fairness disparities in LLMs and proposes practical interventions to reduce subgroup bias and improve response stability.

Findings

Subgroup disparities in response variability are significant before mitigation.

Mitigation strategies reduce cross-group divergence, enhancing fairness.

Prompt sensitivity varies across demographic subgroups, affecting model outputs.

Abstract

Large Language Models (LLMs), though shown to be effective in many applications, can vary significantly in their response quality. In this paper, we investigate this problem of prompt fairness: specifically, the phrasing of a prompt by different users/styles, despite the same question being asked in principle, may elicit different responses from an LLM. To quantify this disparity, we propose to use information-theoretic metrics that can capture two dimensions of bias: subgroup sensitivity, the variability of responses within a subgroup and cross group consistency, the variability of responses across subgroups. Our analysis reveals that certain subgroups exhibit both higher internal variability and greater divergence from others. Our empirical analysis reveals that certain demographic sub groups experience both higher internal variability and greater divergence from others, indicating structural inequities in model behavior. To mitigate these disparities, we propose practical interventions, including majority voting across multiple generations and prompt neutralization, which together improve response stability and enhance fairness across user populations. In the experiments, we observe clear prompt sensitivity disparities across demographic subgroups: before mitigation, cross-group divergence values reach 0.28 and typically fall in the from 0.14 to 0.22 range. After applying our neutralization and multi generation strategy, these divergences consistently decrease, with the largest gap reduced to 0.22 and many distances falling to 0.17 or below, indicating more stable and consistent outputs across subgroups.