Design Considerations in Offline Preference-based RL

TL;DR

This paper provides a theoretical analysis of design choices in offline preference-based reinforcement learning, such as loss functions and data sampling, and verifies some findings empirically on a summarization task.

Contribution

It offers a unified theoretical framework for various offline RLHF methods, highlighting how different design choices impact policy quality.

Findings

Loss function choice significantly affects policy performance.

The policy used for normalization influences learning outcomes.

Data sampling policy plays a crucial role in offline RLHF effectiveness.

Abstract

Offline algorithms for Reinforcement Learning from Human Preferences (RLHF), which use only a fixed dataset of sampled responses given an input, and preference feedback among these responses, have gained increasing prominence in the literature on aligning language models. In this paper, we study how the different design choices made in methods such as DPO, IPO, SLiC and many variants influence the quality of the learned policy, from a theoretical perspective. Our treatment yields insights into the choices of loss function, the policy which is used to normalize log-likelihoods, and also the role of the data sampling policy. Notably, our results do not rely on the standard reparameterization-style arguments used to motivate some of the algorithms in this family, which allows us to give a unified treatment to a broad class of methods. We also conduct a small empirical study to verify some of the theoretical findings on a standard summarization benchmark.