KL Penalty Control via Perturbation for Direct Preference Optimization

TL;DR

This paper introduces $oldsymbol{ ext{ extepsilon}- ext{DPO}}$, an adaptive method for controlling the KL penalty in Direct Preference Optimization, improving alignment of language models with human preferences by adjusting penalties per preference pair.

Contribution

The paper proposes $ ext{ extepsilon}$-DPO, a novel adaptive KL penalty control method that adjusts penalty strength dynamically for each preference pair during training.

Findings

$ ext{ extepsilon}$-DPO significantly improves DPO performance on chatbot benchmarks.

Adaptive KL penalty control reflects preference model confusion and enhances preference confidence.

The method provides an efficient trade-off in KL penalty, leading to better alignment results.

Abstract

Direct Preference Optimization (DPO) demonstrates the advantage of aligning a large language model with human preference using only an offline dataset. However, DPO has the limitation that the KL penalty, which prevents excessive deviation from the reference model, is static throughout the training process. Several methods claim to change this static KL penalty of DPO into a dynamic one, but no approach can adaptively assign different KL penalties for each preference pair. In this paper, we propose $\varepsilon$-Direct Preference Optimization ($\varepsilon$-DPO), which allows adaptive control of the KL penalty strength $\beta$ for each preference pair. Specifically, $\varepsilon$-DPO adaptively controls $\beta$ for each preference pair based on the monotonicity of logits as a preference model under the perturbation of $\beta$ during training. This is equivalent to adjusting the KL penalty by checking whether the change in training-time temperature can lead to better preference confidence as preference models by simply reusing the logit of the current policy and the reference policy. Experimental results show that the simple criterion of $\varepsilon$-DPO for KL penalty relaxation significantly improves DPO compared to most existing direct alignment algorithms on general chatbot benchmarks and reveal that this KL penalty control criterion can reflect confusion as a preference model and provide an efficient KL trade-off, highlighting the significance of instance-level adaptive KL penalty control in DPO.