Structure from Strategic Interaction & Uncertainty: Risk Sensitive Games for Robust Preference Learning

TL;DR

This paper introduces risk-sensitive preference games for robust policy learning in large language models, addressing tail behavior and systematic failures by optimizing convex risk measures.

Contribution

It develops a novel risk-sensitive game framework that retains convergence properties and provides algorithms with stability and sample complexity guarantees.

Findings

Risk-adjusted policies improve robustness across data strata.

The hierarchical game and bias correction method converge to the Stackelberg equilibrium.

Empirical results show risk-sensitive policies outperform risk-neutral ones in stability and robustness.

Abstract

A growing line of work reframes preference-based fine-tuning of large language models game-theoretically: Nash Learning from Human Feedback (NLHF) recasts the problem as a zero-sum game over policies. However, optimization is over expected pairwise payoffs, thereby conflating policies with similar win rates but different tail behavior. As such, these methods are agnostic to where in the data distribution they succeed or fail: strong average performance can mask systematic failure across prompts, annotators, or safety-critical strata. We introduce risk-sensitive preference games, in which players optimize convex risk measures of their preference loss, exploiting structure in preference uncertainty. While risk-sensitivity generally breaks the zero-sum structure, we show that translation invariance of many risk metrics ensures that we retain monotonicity, yielding fast convergence of sample-efficient self-play methods. Furthermore, we establish algorithmic stability and offline sample complexity bounds that scale with risk, requiring simultaneous control of structural bias from nonlinear risk transformations, statistical bias in risk estimation, and concentration tailored to the risk-sensitive setting. To address statistical bias, we introduce a hierarchical game formulation and a two-timescale extragradient algorithm with bias correction that converges to the Stackelberg equilibrium and is especially effective in low-sample regimes. Empirically, risk-adjusted policies are robust across data strata, stable across risk choices, and match or exceed risk-neutral performance thereby achieving robustness without a performance tax.