Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration

TL;DR

This paper introduces SE-POPO, a novel online RLHF algorithm that achieves polynomial sample complexity scaling with reward scale, significantly improving efficiency in preference-based language model alignment tasks.

Contribution

The paper presents SE-POPO, the first online RLHF algorithm with polynomial sample complexity scaling, addressing a key limitation of existing methods.

Findings

SE-POPO outperforms existing algorithms in sample efficiency.

Theoretical analysis shows SE-POPO's sample complexity dominates prior methods.

Empirical results confirm SE-POPO's effectiveness on benchmarks.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for large language model (LLM) alignment. This paper studies the setting of online RLHF and focus on improving sample efficiency. All existing algorithms in online RLHF, whether doing passive exploration or active exploration, suffer from a sample complexity that scales exponentially with the scale of the reward function. This fundamental limitation hinders their effectiveness in scenarios with heavily skewed preferences, e.g. questions with a unique correct solution. To address this, we introduce Self-Exploring Preference-Incentive Online Preference Optimization (SE-POPO), an online RLHF algorithm that for the first time achieves a sample complexity that scales polynomially with the reward scale, answering an open problem raised by Xie et al. (2024).. Theoretically, we demonstrate that the sample complexity of SE-POPO dominates that of existing exploration algorithms. Empirically, our systematic evaluation confirms that SE-POPO is more sample-efficient than both exploratory and non-exploratory baselines, in two primary application scenarios of RLHF as well as on public benchmarks, marking a significant step forward in RLHF algorithm design. The code is available at https://github.com/MYC000801/SE-POPO.