Thompson Sampling in Online RLHF with General Function Approximation

TL;DR

This paper introduces a Thompson sampling-based algorithm for online reinforcement learning from human feedback with function approximation, providing theoretical regret guarantees and new concentration inequalities.

Contribution

It develops a model-free posterior sampling method for online RLHF with theoretical regret bounds using Bellman eluder dimension, advancing understanding of RLHF efficiency.

Findings

Achieves $O(\sqrt{T})$ regret bound for the proposed algorithm.

Establishes a new concentration inequality for squared Bellman error based on MLE.

Provides theoretical analysis linking function class complexity to regret in RLHF.

Abstract

Reinforcement learning from human feedback (RLHF) has achieved great empirical success in aligning large language models (LLMs) with human preference, and it is of great importance to study the statistical efficiency of RLHF algorithms from a theoretical perspective. In this work, we consider the online RLHF setting where the preference data is revealed during the learning process and study action value function approximation. We design a model-free posterior sampling algorithm for online RLHF inspired by Thompson sampling and provide its theoretical guarantee. Specifically, we adopt Bellman eluder (BE) dimension as the complexity measure of the function class and establish $O(\sqrt{T})$ regret bound for the proposed algorithm with other multiplicative factor depending on the horizon, BE dimension and the $log$-bracketing number of the function class. Further, in the analysis, we first establish the concentration-type inequality of the squared Bellman error bound based on the maximum likelihood estimator (MLE) generalization bound, which plays the crucial rules in obtaining the eluder-type regret bound and may be of independent interest.