ComPO: Preference Alignment via Comparison Oracles

TL;DR

This paper introduces ComPO, a novel preference alignment method using comparison oracles that effectively handles noisy preference pairs, improving LLM alignment with human preferences through a zeroth-order optimization approach.

Contribution

Proposes a new comparison-based preference alignment method with convergence guarantees and demonstrates its effectiveness on multiple models and benchmarks.

Findings

Effective in improving LLM performance with noisy preference data

Outperforms existing direct alignment methods in experiments

Highlights the importance of specialized methods for different preference pair likelihood margins

Abstract

Direct alignment methods are increasingly used for aligning large language models (LLMs) with human preferences. However, these methods suffer from the issues of verbosity and likelihood displacement, which can be driven by the noisy preference pairs that induce similar likelihood for preferred and dispreferred responses. The contributions of this paper are two-fold. First, we propose a new preference alignment method based on zeroth-order, comparison-based optimization via comparison oracles and provide convergence guarantees for its basic scheme. Second, we improve our method using some heuristics and conduct the experiments to demonstrate the flexibility and compatibility of practical scheme in improving the performance of LLMs using noisy preference pairs. Evaluations are conducted across multiple base and instruction-tuned models (Mistral-7B, Llama-3-8B and Gemma-2-9B) with benchmarks (AlpacaEval 2, MT-Bench and Arena-Hard). Experimental results show the effectiveness of our method as an alternative to addressing the limitations of existing direct alignment methods. A highlight of our work is that we evidence the importance of designing specialized methods for preference pairs with distinct likelihood margin, which complements the recent findings in Razin et al (2025).