Mind the Gap: Structure-Aware Consistency in Preference Learning

TL;DR

This paper addresses the inconsistency of standard surrogate losses in preference learning for LLM alignment, proposing a structure-aware framework with theoretical guarantees and a novel objective to improve consistency.

Contribution

It introduces a margin-shifted ranking framework with structure-aware consistency bounds and a new objective (SA-DPO) that adapts margins based on semantic distances.

Findings

Standard surrogates are inconsistent for neural network hypothesis sets.

The proposed SA-DPO adapts margins to semantic distances, improving alignment.

Heavy-tailed surrogates outperform logistic loss in capacity-bounded models.

Abstract

Preference learning has become the foundation of aligning Large Language Models (LLMs) with human intent. Popular methods, such as Direct Preference Optimization (DPO), minimize surrogate losses as proxies for the intractable pairwise ranking loss. However, we demonstrate that for the equicontinuous hypothesis sets typical of neural networks, these standard surrogates are theoretically inconsistent, yielding vacuous generalization guarantees. To resolve this, we formulate LLM alignment within a margin-shifted ranking framework. We derive rigorous $H$-consistency bounds that depend on enforcing a separation margin $\gamma$. Crucially, we extend this to Structure-Aware $H$-consistency, introducing a novel objective (SA-DPO) that adapts the margin based on the semantic distance between responses to handle synonyms and hard pairs. Finally, we analyze the trade-off between consistency and model limitations via the Margin-Capacity Profile, proving that heavy-tailed surrogates (such as the Polynomial Hinge family) offer superior consistency guarantees for capacity-bounded models compared to the standard logistic loss used in DPO.