Small-Margin Preferences Still Matter-If You Train Them Right

TL;DR

This paper introduces MixDPO, a difficulty-aware training method for preference-based alignment of large language models, effectively leveraging ambiguous pairs by combining preference loss and supervised fine-tuning to improve alignment performance.

Contribution

The paper proposes MixDPO, a novel curriculum-based hybrid training strategy that improves preference-based model alignment by handling difficult pairs more effectively.

Findings

MixDPO outperforms DPO and variants on multiple benchmarks.

It achieves significant gains on AlpacaEval 2 length-controlled win rate.

Difficulty-aware training stabilizes preference-based alignment methods.

Abstract

Preference optimization methods such as DPO align large language models (LLMs) using paired comparisons, but their effectiveness can be highly sensitive to the quality and difficulty of preference pairs. A common heuristic treats small-margin (ambiguous) pairs as noisy and filters them out. In this paper, we revisit this assumption and show that pair difficulty interacts strongly with the optimization objective: when trained with preference-based losses, difficult pairs can destabilize training and harm alignment, yet these same pairs still contain useful supervision signals when optimized with supervised fine-tuning (SFT). Motivated by this observation, we propose MixDPO, a simple yet effective difficulty-aware training strategy that (i) orders preference data from easy to hard (a curriculum over margin-defined difficulty), and (ii) routes difficult pairs to an SFT objective while applying a preference loss to easy pairs. This hybrid design provides a practical mechanism to leverage ambiguous pairs without incurring the optimization failures often associated with preference losses on low-margin data. Across three LLM-judge benchmarks, MixDPO consistently improves alignment over DPO and a range of widely-used variants, with particularly strong gains on AlpacaEval~2 length-controlled (LC) win rate.