Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game

TL;DR

This paper introduces SLHF, a sequential game framework for preference optimization in AI alignment, enabling iterative refinement and improved robustness over existing methods like RLHF and NLHF.

Contribution

SLHF models preference optimization as a sequential game, capturing richer preferences and enabling inference-time refinements, which improves alignment and robustness.

Findings

SLHF achieves strong alignment across diverse datasets.

SLHF scales from 0.5B to 8B parameters.

Inference-time refinements transfer across models.

Abstract

We introduce Stackelberg Learning from Human Feedback (SLHF), a new framework for preference optimization. SLHF frames the alignment problem as a sequential-move game between two policies: a Leader, which commits to an action, and a Follower, which responds conditionally on the Leader's action. This approach decomposes preference optimization into a refinement problem for the Follower and an optimization problem against an adversary for the Leader. Unlike Reinforcement Learning from Human Feedback (RLHF), which assigns scalar rewards to actions, or Nash Learning from Human Feedback (NLHF), which seeks a simultaneous-move equilibrium, SLHF leverages the asymmetry of sequential play to capture richer preference structures. The sequential design of SLHF naturally enables inference-time refinement, as the Follower learns to improve the Leader's actions, and these refinements can be leveraged through iterative sampling. We compare the solution concepts of SLHF, RLHF, and NLHF, and lay out key advantages in consistency, data sensitivity, and robustness to intransitive preferences. Experiments on large language models demonstrate that SLHF achieves strong alignment across diverse preference datasets, scales from 0.5B to 8B parameters, and yields inference-time refinements that transfer across model families without further fine-tuning.