A Practical Analysis of Human Alignment with *PO

TL;DR

This paper empirically evaluates the robustness of human alignment methods like DPO and introduces LN-DPO, a length-normalized variant that enhances stability and performance across hyperparameters in realistic out-of-distribution scenarios.

Contribution

It introduces LN-DPO, a simple length-normalized method, and provides a comprehensive analysis of the robustness of state-of-the-art human alignment techniques under varying hyperparameters.

Findings

LN-DPO reduces average response length and improves stability.

State-of-the-art methods perform similarly at their peak.

Performance patterns vary significantly away from optimal conditions.

Abstract

At the forefront of state-of-the-art human alignment methods are preference optimization methods (*PO). Prior research has often concentrated on identifying the best-performing method, typically involving a grid search over hyperparameters, which can be impractical for general practitioners. In this paper, we examine the robustness of existing state-of-the-art methods to varying hyperparameters in a realistic out-of-distribution (OOD) scenario that mirrors real-world applications of human alignment. Our goal is to empirically find the method that increases the likelihood of achieving better results through the lens of various metrics, such as KL divergence and response length. We also introduce LN-DPO, a simple length-normalized version of DPO that is more stable across hyperparameters, effectively reduces the average response length, and improves performance. Our analysis of state-of-the-art reference-free (i.e., SimPO) and reference-dependent (i.e., DPO and LN-DPO) methods reveals that they perform similarly at their peak (i.e., best possible scenario). However, we uncover that the pattern of change in performance greatly varies as we move away from the best possible scenario.