Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Model

TL;DR

This paper introduces a semiparametric single-index model for preference alignment in large language models, addressing unknown link functions and developing robust algorithms with convergence guarantees, demonstrated empirically.

Contribution

It proposes a novel semiparametric model for preference alignment that handles unknown link functions and develops robust algorithms with theoretical guarantees.

Findings

Algorithms converge with guarantees under generic function complexity.

Empirical results demonstrate effectiveness on LLM alignment.

Model handles unidentifiable nonparametric indices effectively.

Abstract

Aligning large language models (LLMs) to preference data typically assumes a known link function between observed preferences and latent rewards (e.g., a logistic Bradley-Terry link). Misspecification of this link can bias inferred rewards and misalign learned policies. We study preference alignment under an unknown and unrestricted link function. We show that realizability of $f$-divergence-constrained reward maximization in a policy class induces a semiparametric single-index binary choice model, where a scalar policy-dependent index captures all dependence on demonstrations and the remaining preference distribution is unrestricted. Rather than assuming this model has identifiable finite-dimensional structural parameters and estimating them, as in econometrics, we focus on policy learning with the reward function implicit, analyzing error to the optimal policy and allowing for unidentifiable nonparametric indices. We develop preference optimization algorithms robust to the unknown link and prove convergence guarantees in terms of generic function complexity measures. We demonstrate this empirically on LLM alignment. Code is available at https://github.com/causalml/spo/