Hedging and Non-Affirmation: Quantifying LLM Alignment on Questions of Human Rights

TL;DR

This paper introduces a framework to quantify and analyze hedging and non-affirmation behaviors in large language models regarding human rights across various identity groups, revealing significant identity-dependent disparities.

Contribution

It systematically measures these behaviors across multiple models and identities, and demonstrates that group steering effectively reduces biased responses.

Findings

4 out of 7 models show identity-dependent hedging behaviors

Identity is the strongest predictor of hedging and non-affirmation behaviors

Group steering effectively mitigates these behaviors across query types

Abstract

Hedging and non-affirmation are behaviors exhibited by large language models (LLMs) that limit the clear endorsement of specific statements. While these behaviors are desirable in subjective contexts, they are undesirable in the context of human rights - which apply unambiguously to all groups. We present a systematic framework to measure these behaviors in unconstrained LLM responses regarding various identity groups. We evaluate six large proprietary models as well as one open-weight LLM on 4738 prompts across 205 national and stateless ethnic identities and find that 4 out of 7 display hedging and non-affirmation that is significantly dependent on the identity of the group. While factors like conflict signals, sovereignty (whether identity is stateless), or economic indicators (GDP) also influence model behavior, their effect sizes are consistently weaker than the impact of identity itself. The systematic disparity is robust to methods of rephrasing the prompts. Since group identity is the strongest predictor of these behaviors, we use open-weight models to explore whether applying steering and orthogonalization techniques to these group identities can mitigate the rates of hedging and non-affirmation behaviors. We find that group steering is the most effective debiasing approach across query types and is robust to downstream forgetting.