Understanding the Logic of Direct Preference Alignment through Logic

TL;DR

This paper introduces a formal framework for understanding and deriving direct preference alignment losses in large language models, providing insights into their semantics, relationships, and potential for systematic development.

Contribution

It formalizes DPA losses using symbolic reasoning, enabling systematic derivation and analysis of various preference alignment algorithms.

Findings

Symbolic formalism characterizes common DPA variants.

Framework reveals relationships between different DPA losses.

Enables systematic exploration and creation of new loss functions.

Abstract

Recent direct preference alignment algorithms (DPA), such as DPO, have shown great promise in aligning large language models to human preferences. While this has motivated the development of many new variants of the original DPO loss, understanding the differences between these recent proposals, as well as developing new DPA loss functions, remains difficult given the lack of a technical and conceptual framework for reasoning about the underlying semantics of these algorithms. In this paper, we attempt to remedy this by formalizing DPA losses in terms of discrete reasoning problems. Specifically, we ask: Given an existing DPA loss, can we systematically derive a symbolic program that characterizes its semantics? We propose a novel formalism for characterizing preference losses for single model and reference model based approaches, and identify symbolic forms for a number of commonly used DPA variants. Further, we show how this formal view of preference learning sheds new light on both the size and structure of the DPA loss landscape, making it possible to not only rigorously characterize the relationships between recent loss proposals but also to systematically explore the landscape and derive new loss functions from first principles. We hope our framework and findings will help provide useful guidance to those working on human AI alignment.