Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization

TL;DR

This paper enhances preference optimization for large language models by improving data generation with an iterative ranking method and introducing a budget-controlled regularization to achieve better alignment, outperforming current state-of-the-art methods.

Contribution

It proposes an iterative pairwise ranking mechanism for high-quality preference data and a novel budget-controlled regularization to improve preference optimization stability and effectiveness.

Findings

Iterative ranking improves preference data quality.

Budget-controlled regularization enhances convergence.

Models surpass SOTA on benchmark tasks.

Abstract

Direct Preference Optimization (DPO) and its variants have become the de facto standards for aligning large language models (LLMs) with human preferences or specific goals. However, DPO requires high-quality preference data and suffers from unstable preference optimization. In this work, we aim to improve the preference optimization pipeline by taking a closer look at preference data generation and training regularization techniques. For preference data generation, we demonstrate that existing scoring-based reward models produce unsatisfactory preference data and perform poorly on out-of-distribution tasks. This significantly impacts the LLM alignment performance when using these data for preference tuning. To ensure high-quality preference data generation, we propose an iterative pairwise ranking mechanism that derives preference ranking of completions using pairwise comparison signals. For training regularization, we observe that preference optimization tends to achieve better convergence when the LLM predicted likelihood of preferred samples gets slightly reduced. However, the widely used supervised next-word prediction regularization strictly prevents any likelihood reduction of preferred samples. This observation motivates our design of a budget-controlled regularization formulation. Empirically we show that combining the two designs leads to aligned models that surpass existing SOTA across two popular benchmarks.