Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective

TL;DR

This paper provides a theoretical analysis of Direct Preference Optimization (DPO), revealing its limitations in balancing preferred and dispreferred responses, and offers insights for future improvements of LLM alignment methods.

Contribution

It introduces a field theory-based analytical framework to understand DPO's optimization process and its limitations in aligning models with human preferences.

Findings

DPO decreases dispreferred data probability faster than preferred data.

Theoretical insights explain DPO's sensitivity and learning capacity issues.

Framework sets foundation for improving DPO in LLM alignment.

Abstract

Direct Preference Optimization (DPO), which derives reward signals directly from pairwise preference data, has shown its effectiveness on aligning Large Language Models (LLMs) with human preferences. Despite its widespread use across various tasks, DPO has been criticized for its sensitivity to the SFT's effectiveness and its hindrance to the learning capacity towards human-preferred responses, leading to less satisfactory performance. To overcome those limitations, the theoretical understanding of DPO are indispensable but still lacking. To this end, we take a step towards theoretically analyzing and understanding the limitations of DPO. Specifically, we provide an analytical framework using the field theory to analyze the optimization process of DPO. By analyzing the gradient vector field of the DPO loss function, we find that the DPO loss function decreases the probability of producing human dispreferred data at a faster rate than it increases the probability of producing preferred data. This provides theoretical insights for understanding the limitations of DPO discovered in the related research experiments, thereby setting the foundation for its improvement.