Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment

TL;DR

This paper introduces a unified mathematical framework called Reward-Aware Preference Optimization (RPO) that consolidates various LLM preference optimization methods, enabling systematic analysis and practical guidance for model alignment improvements.

Contribution

The paper presents RPO, a comprehensive framework that unifies existing preference optimization techniques and facilitates systematic study of their design choices in LLM alignment.

Findings

RPO effectively unifies multiple preference optimization methods.

Systematic ablation studies reveal key factors influencing LLM alignment.

Practical guidance for improving model alignment strategies.

Abstract

The rapid development of large language model (LLM) alignment algorithms has resulted in a complex and fragmented landscape, with limited clarity on the effectiveness of different methods and their inter-connections. This paper introduces Reward-Aware Preference Optimization (RPO), a mathematical framework that unifies popular preference optimization techniques in LLM alignment, including DPO, IPO, SimPO, and REINFORCE (LOO), among others. RPO provides a structured approach to disentangle and systematically study the impact of various design choices, such as the optimization objective, the number of responses per prompt, and the use of implicit versus explicit reward models, on LLM preference optimization. We additionally propose a new experimental setup that enables the clean and direct ablation of such design choices. Through an extensive series of ablation studies within the RPO framework, we gain insights into the critical factors shaping model alignment, offering practical guidance on the most effective strategies for improving LLM alignment.