Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information

TL;DR

This paper introduces a mutual information-based unifying framework for preference optimization in large language models, simplifying understanding of various DPO variants and aiding future alignment research.

Contribution

The paper proposes a new flexible loss function framework that unifies many existing DPO algorithms through specified priors, enhancing interpretability and development of LLM alignment methods.

Findings

Many DPO variants can be derived from the proposed framework

The framework clarifies relationships between different alignment algorithms

Potential for developing more robust and interpretable alignment techniques

Abstract

Post-alignment of large language models (LLMs) is critical in improving their utility, safety, and alignment with human intentions. Direct preference optimisation (DPO) has become one of the most widely used algorithms for achieving this alignment, given its ability to optimise models based on human feedback directly. However, the vast number of DPO variants in the literature has made it increasingly difficult for researchers to navigate and fully grasp the connections between these approaches. This paper introduces a unifying framework inspired by mutual information, which proposes a new loss function with flexible priors. By carefully specifying these priors, we demonstrate that many existing algorithms, such as SimPO, TDPO, SparsePO, and others, can be derived from our framework. This unification offers a clearer and more structured approach, allowing researchers to understand the relationships between different DPO variants better. We aim to simplify the landscape of DPO algorithms, making it easier for the research community to gain insights and foster further advancements in LLM alignment. Ultimately, we hope our framework can be a foundation for developing more robust and interpretable alignment techniques.