Minor DPO reject penalty to increase training robustness

TL;DR

This paper analyzes the Direct Preference Optimization (DPO) method for LLM fine-tuning, identifies its limitations, and proposes MinorDPO to improve alignment and training stability by adjusting the penalty term.

Contribution

The paper introduces MinorDPO, a modification to DPO that enhances alignment with RL algorithms and increases training robustness in preference-based LLM fine-tuning.

Findings

MinorDPO improves training stability.

MinorDPO better aligns with RL algorithms.

Analysis of DPO's $eta$ parameter reveals its impact.

Abstract

Learning from human preference is a paradigm used in large-scale language model (LLM) fine-tuning step to better align pretrained LLM to human preference for downstream task. In the past it uses reinforcement learning from human feedback (RLHF) algorithm to optimize the LLM policy to align with these preferences and not to draft too far from the original model. Recently, Direct Preference Optimization (DPO) has been proposed to solve the alignment problem with a simplified RL-free method. Using preference pairs of chosen and reject data, DPO models the relative log probability as implicit reward function and optimize LLM policy using a simple binary cross entropy objective directly. DPO is quite straight forward and easy to be understood. It perform efficiently and well in most cases. In this article, we analyze the working mechanism of $\beta$ in DPO, disclose its syntax difference between RL algorithm and DPO, and understand the potential shortage brought by the DPO simplification. With these insights, we propose MinorDPO, which is better aligned to the original RL algorithm, and increase the stability of preference optimization process.