Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective

TL;DR

This paper introduces FiSCo, a semantic and statistical framework for evaluating group fairness in LLMs by analyzing long-form responses at the claim level, effectively detecting subtle biases beyond token-level methods.

Contribution

FiSCo is a novel framework that assesses fairness in LLMs through semantic claim comparison and hypothesis testing, addressing limitations of existing token-based bias evaluation methods.

Findings

FiSCo reliably detects nuanced biases in LLM outputs.

It reduces false positives caused by output variability.

Outperforms existing evaluation metrics in bias detection.

Abstract

Large Language Models (LLMs) often generate responses with inherent biases, undermining their reliability in real-world applications. Existing evaluation methods often overlook biases in long-form responses and the intrinsic variability of LLM outputs. To address these challenges, we propose FiSCo (Fine-grained Semantic Comparison), a novel statistical framework to evaluate group-level fairness in LLMs by detecting subtle semantic differences in long-form responses across demographic groups. Unlike prior work focusing on sentiment or token-level comparisons, FiSCo goes beyond surface-level analysis by operating at the claim level, leveraging entailment checks to assess the consistency of meaning across responses. We decompose model outputs into semantically distinct claims and apply statistical hypothesis testing to compare inter- and intra-group similarities, enabling robust detection of subtle biases. We formalize a new group counterfactual fairness definition and validate FiSCo on both synthetic and human-annotated datasets spanning gender, race, and age. Experiments show that FiSCo more reliably identifies nuanced biases while reducing the impact of stochastic LLM variability, outperforming various evaluation metrics.