The Differences Between Direct Alignment Algorithms are a Blur

TL;DR

This paper systematically compares Direct Alignment Algorithms (DAAs) for LLMs, revealing that the ranking objective is the main factor influencing alignment quality, overshadowing differences in scalar scoring methods.

Contribution

It introduces a unified training framework for DAAs and highlights the ranking objective as the key determinant of alignment performance.

Findings

Ranking objective is the primary driver of alignment quality.

Scalar score differences are secondary to the ranking objective.

Unified framework improves understanding of DAA performance.

Abstract

Direct Alignment Algorithms (DAAs) simplify LLM alignment by directly optimizing policies, bypassing reward modeling and RL. While DAAs differ in their use of SFT (one-stage vs. two-stage) and the scalar score they optimize (likelihood vs. odds ratios), the key performance drivers remain underexplored. We present a systematic comparison and analyze a previously overlooked axis - the ranking objective (pairwise vs. pointwise). To isolate this factor, we propose a unified training framework across DAAs by (i) converting one-stage methods (ORPO, ASFT) into a two-stage pipeline with an explicit SFT phase and (ii) introducing a $\beta$ parameter that places all methods in the same hyperparameter space and improves the quality of odds-ratio DAAs (ORPO, ASFT). Under this setup, the ranking objective emerges as the primary determinant of alignment quality, whereas the particular scalar score (policy-reference ratio vs. odds ratio) is secondary. We corroborate this on instruction-following tasks and further confirm it on math-reasoning benchmarks across model scales. Evidence suggests that this stems from how these objectives interact with prompt-specific biases, supported both by strictly controlled experiments and by observations on real data. Our findings underscore the need for nuanced evaluations in DAA research to avoid oversimplified claims of superiority.