Human Alignment of Large Language Models through Online Preference Optimisation

TL;DR

This paper demonstrates the equivalence between two recent human alignment methods for large language models, introduces a generalized algorithm IPO-MD leveraging online preference data, and compares its performance with existing methods on summarization tasks.

Contribution

It proves the equivalence between IPO and Nash-MD methods, introduces IPO-MD as a new generalized online alignment algorithm, and evaluates its effectiveness against other online preference optimization techniques.

Findings

IPO and Nash-MD are equivalent when considering online data.

IPO-MD outperforms some existing online preference methods.

The equivalence enables new insights into online human alignment strategies.

Abstract

Ensuring alignment of language models' outputs with human preferences is critical to guarantee a useful, safe, and pleasant user experience. Thus, human alignment has been extensively studied recently and several methods such as Reinforcement Learning from Human Feedback (RLHF), Direct Policy Optimisation (DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In this paper, our contribution is two-fold. First, we show the equivalence between two recent alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror Descent (Nash-MD). Second, we introduce a generalisation of IPO, named IPO-MD, that leverages the regularised sampling approach proposed by Nash-MD. This equivalence may seem surprising at first sight, since IPO is an offline method whereas Nash-MD is an online method using a preference model. However, this equivalence can be proven when we consider the online version of IPO, that is when both generations are sampled by the online policy and annotated by a trained preference model. Optimising the IPO loss with such a stream of data becomes then equivalent to finding the Nash equilibrium of the preference model through self-play. Building on this equivalence, we introduce the IPO-MD algorithm that generates data with a mixture policy (between the online and reference policy) similarly as the general Nash-MD algorithm. We compare online-IPO and IPO-MD to different online versions of existing losses on preference data such as DPO and SLiC on a summarisation task.