Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?

TL;DR

This paper investigates the effectiveness of on-policy versus static preference data in language model alignment, revealing that on-policy data is not always optimal and proposing a stage-based framework for better alignment strategies.

Contribution

It introduces the alignment stage assumption, dividing the process into preference injection and fine-tuning stages, and develops an algorithm to identify optimal data boundaries for improved LM alignment.

Findings

On-policy data can be significantly more or less effective than static data depending on the model.

The proposed boundary measurement algorithm effectively identifies the transition point between alignment stages.

Experimental results across multiple models and methods validate the generality of the alignment stage assumption.

Abstract

The alignment of language models~(LMs) with human preferences is critical for building reliable AI systems. The problem is typically framed as optimizing an LM policy to maximize the expected reward that reflects human preferences. Recently, Direct Preference Optimization~(DPO) was proposed as a LM alignment method that directly optimize the policy from static preference data, and further improved by incorporating on-policy sampling~(i.e., preference candidates generated during the training loop) for better LM alignment. However, we show on-policy data is not always optimal, with systematic effectiveness difference emerging between static and on-policy preference candidates. For example, on-policy data can result in a $3\times$ effectiveness compared with static data for Llama-3, and a $0.4\times$ effectiveness for Zephyr. To explain the phenomenon, we propose the alignment stage assumption, which divides the alignment process into two distinct stages: the preference injection stage, which benefits from diverse data, and the preference fine-tuning stage, which favors high-quality data. Through theoretical and empirical analysis, we characterize these stages and propose an effective algorithm to identify the boundaries between them. We perform experiments on $5$ models~(Llama, Zephyr, Phi-2, Qwen, Pythia) and $2$ alignment methods~(DPO, SLiC-HF) to show the generalizability of alignment stage assumption and the effectiveness of the boundary measurement algorithm.