Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment

TL;DR

This paper introduces a robust, data-efficient framework for aligning large language models with human preferences using a Stackelberg game approach, significantly reducing the need for extensive human-labeled data.

Contribution

The paper proposes SGPO, a Stackelberg game-based alignment method, and SSAPO, a self-annotation technique that achieves strong performance with minimal human labels.

Findings

SSAPO uses only 2K seed preferences to outperform benchmarks.

SSAPO maintains robustness against noisy self-labels.

The approach reduces human annotation costs by over 95%.

Abstract

Aligning large language models (LLMs) with human preferences typically demands vast amounts of meticulously curated data, which is both expensive and prone to labeling noise. We propose Stackelberg Game Preference Optimization (SGPO), a robust alignment framework that models alignment as a two-player Stackelberg game between a policy (leader) and a worst-case preference distribution (follower). The proposed SGPO guarantees $\mathcal{O}(\epsilon)$-bounded regret within an $\epsilon$-Wasserstein ball, offering formal robustness to (self-)annotation noise. We instantiate SGPO with Stackelberg Self-Annotated Preference Optimization (SSAPO), which uses minimal human-labeled "seed" preferences and iteratively self-annotates new prompts. In each iteration, SSAPO applies a distributionally robust reweighting of synthetic annotations, ensuring that noisy or biased self-labels do not derail training. Remarkably, using only 2K seed preferences -- about 1/30 of standard human labels -- SSAPO achieves strong win rates against GPT-4 across multiple benchmarks within three iterations. These results highlight that a principled Stackelberg formulation yields data-efficient alignment for LLMs, significantly reducing reliance on costly human annotations.