Entropy Controllable Direct Preference Optimization

TL;DR

This paper introduces H-DPO, an entropy-controllable modification of DPO, improving policy mode-seeking in LLM training by adjusting entropy, leading to better performance in mathematical tasks.

Contribution

H-DPO provides a simple way to control policy entropy in DPO, enhancing mode-seeking and performance without complex changes.

Findings

H-DPO outperforms DPO in pass@k evaluations for math tasks.

H-DPO is easy to implement with minor loss modifications.

Entropy control improves mode coverage in policy training.

Abstract

In the post-training of large language models (LLMs), Reinforcement Learning from Human Feedback (RLHF) is an effective approach to achieve generation aligned with human preferences. Direct Preference Optimization (DPO) allows for policy training with a simple binary cross-entropy loss without a reward model. The objective of DPO is regularized by reverse KL divergence that encourages mode-seeking fitting to the reference policy. Nonetheless, we indicate that minimizing reverse KL divergence could fail to capture a mode of the reference distribution, which may hurt the policy's performance. Based on this observation, we propose a simple modification to DPO, H-DPO, which allows for control over the entropy of the resulting policy, enhancing the distribution's sharpness and thereby enabling mode-seeking fitting more effectively. In our experiments, we show that H-DPO outperformed DPO across various tasks, demonstrating superior results in pass@$k$ evaluations for mathematical tasks. Moreover, H-DPO is simple to implement, requiring only minor modifications to the loss calculation of DPO, which makes it highly practical and promising for wide-ranging applications in the training of LLMs.