On Extending Direct Preference Optimization to Accommodate Ties

TL;DR

This paper extends Direct Preference Optimization (DPO) to explicitly model ties in pairwise comparisons, leading to improved regularization and performance in tasks like translation and summarization.

Contribution

The paper introduces two DPO variants that incorporate ties using Rao-Kupper and Davidson models, demonstrating their benefits over standard DPO.

Findings

Explicitly modeling ties improves task performance.

Including ties enhances regularization measured by KL divergence.

Performance gains observed in translation and reasoning tasks.

Abstract

We derive and investigate two DPO variants that explicitly model the possibility of declaring a tie in pair-wise comparisons. We replace the Bradley-Terry model in DPO with two well-known modeling extensions, by Rao and Kupper and by Davidson, that assign probability to ties as alternatives to clear preferences. Our experiments in neural machine translation and summarization show that explicitly labeled ties can be added to the datasets for these DPO variants without the degradation in task performance that is observed when the same tied pairs are presented to DPO. We find empirically that the inclusion of ties leads to stronger regularization with respect to the reference policy as measured by KL divergence, and we see this even for DPO in its original form. We provide a theoretical explanation for this regularization effect using ideal DPO policy theory. We further show performance improvements over DPO in translation and mathematical reasoning using our DPO variants. We find it can be beneficial to include ties in preference optimization rather than simply discard them, as is done in common practice.