AIPO: Improving Training Objective for Iterative Preference Optimization

TL;DR

This paper introduces AIPO, a new training objective for iterative preference optimization that addresses length exploitation issues, leading to state-of-the-art results in aligning large language models.

Contribution

We propose Agreement-aware Iterative Preference Optimization (AIPO), a novel training objective that improves iterative preference optimization for aligning large language models.

Findings

AIPO achieves state-of-the-art performance on MT-Bench.

AIPO effectively mitigates length exploitation in iterative preference optimization.

Experimental results demonstrate the superiority of AIPO over existing methods.

Abstract

Preference Optimization (PO), is gaining popularity as an alternative choice of Proximal Policy Optimization (PPO) for aligning Large Language Models (LLMs). Recent research on aligning LLMs iteratively with synthetic or partially synthetic data shows promising results in scaling up PO training for both academic settings and proprietary trained models such as Llama3. Despite its success, our study shows that the length exploitation issue present in PO is even more severe in Iterative Preference Optimization (IPO) due to the iterative nature of the process. In this work, we study iterative preference optimization with synthetic data. We share the findings and analysis along the way of building the iterative preference optimization pipeline. More specifically, we discuss the length exploitation issue during iterative preference optimization and propose our training objective for iterative preference optimization, namely Agreement-aware Iterative Preference Optimization (AIPO). To demonstrate the effectiveness of our method, we conduct comprehensive experiments and achieve state-of-the-art performance on MT-Bench, AlpacaEval 2.0, and Arena-Hard. Our implementation and model checkpoints will be made available at https://github.com/bytedance/AIPO.