Equal Access, Unequal Interaction: A Counterfactual Audit of LLM Fairness

TL;DR

This study investigates fairness in large language models by examining how they differ in tone, uncertainty, and framing across demographic identities after access is granted, revealing disparities in interaction quality.

Contribution

The paper introduces a counterfactual audit method to evaluate interaction-level fairness in LLMs, highlighting disparities beyond access-based fairness assessments.

Findings

GPT-4 shows more hedging toward younger male users.

LLaMA exhibits broader sentiment variation across identities.

Both models have zero refusal rates, indicating equal access.

Abstract

Prior work on fairness in large language models (LLMs) has primarily focused on access-level behaviors such as refusals and safety filtering. However, equitable access does not ensure equitable interaction quality once a response is provided. In this paper, we conduct a controlled fairness audit examining how LLMs differ in tone, uncertainty, and linguistic framing across demographic identities after access is granted. Using a counterfactual prompt design, we evaluate GPT-4 and LLaMA-3.1-70B on career advice tasks while varying identity attributes along age, gender, and nationality. We assess access fairness through refusal analysis and measure interaction quality using automated linguistic metrics, including sentiment, politeness, and hedging. Identity-conditioned differences are evaluated using paired statistical tests. Both models exhibit zero refusal rates across all identities, indicating uniform access. Nevertheless, we observe systematic, model-specific disparities in interaction quality: GPT-4 expresses significantly higher hedging toward younger male users, while LLaMA exhibits broader sentiment variation across identity groups. These results show that fairness disparities can persist at the interaction level even when access is equal, motivating evaluation beyond refusal-based audits.