Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration

TL;DR

This paper introduces Hybrid Preference Optimization (HPO), a method that combines offline human preference data with online exploration to improve the efficiency and convergence of reinforcement learning from human feedback, with proven theoretical guarantees.

Contribution

The paper presents the first provably optimal hybrid RLHF algorithm that relaxes offline data requirements and enhances online exploration efficiency with matching lower bounds.

Findings

Improved sample efficiency over pure offline and online methods

Provably optimal theoretical bounds for hybrid RLHF

Relaxed offline data requirements for better exploration

Abstract

Reinforcement Learning from Human Feedback (RLHF) is currently the leading approach for aligning large language models with human preferences. Typically, these models rely on extensive offline preference datasets for training. However, offline algorithms impose strict concentrability requirements, which are often difficult to satisfy. On the other hand, while online algorithms can avoid the concentrability issue, pure online exploration could be expensive due to the active preference query cost and real-time implementation overhead. In this paper, we propose a novel approach: Hybrid Preference Optimization (HPO) which combines online exploration with existing offline preferences by relaxing the stringent concentrability conditions for offline exploration, as well as significantly improving the sample efficiency for its online counterpart. We give the first provably optimal theoretical bound for Hybrid RLHF with preference feedback, providing sample complexity bounds for policy optimization with matching lower bounds. Our results yield improved sample efficiency of hybrid RLHF over pure offline and online exploration.