Provably Efficient Online RLHF with One-Pass Reward Modeling

TL;DR

This paper introduces a one-pass reward modeling algorithm for online RLHF that avoids storing historical data, enabling constant-time updates and improving efficiency in aligning language models with human preferences.

Contribution

It formalizes online RLHF as a preference bandit problem and develops a novel online mirror descent algorithm with theoretical guarantees, reducing computational costs.

Findings

Achieves constant-time updates per iteration.

Demonstrates improved statistical and computational efficiency.

Validates effectiveness on large language models with real datasets.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has shown remarkable success in aligning Large Language Models (LLMs) with human preferences. Traditional RLHF methods rely on a fixed dataset, which often suffers from limited coverage. To this end, online RLHF has emerged as a promising direction, enabling iterative data collection and refinement. Despite its potential, this paradigm faces a key bottleneck: the requirement to continuously integrate new data into the dataset and re-optimize the model from scratch at each iteration, resulting in computational and storage costs that grow linearly with the number of iterations. In this work, we address this challenge by proposing a one-pass reward modeling method that eliminates the need to store historical data and achieves constant-time updates per iteration. Specifically, we first formalize RLHF as a contextual preference bandit and develop a new algorithm based on online mirror descent with a tailored local norm, replacing the standard maximum likelihood estimation for reward modeling. We then apply it to various online RLHF settings, including passive data collection, active data collection, and deployment-time adaptation. We provide theoretical guarantees showing that our method enhances both statistical and computational efficiency. Finally, we design practical algorithms for LLMs and conduct experiments with the Llama-3-8B-Instruct and Qwen2.5-7B-Instruct models on Ultrafeedback and Mixture2 datasets, validating the effectiveness of our approach.