$f$-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses

TL;DR

This paper develops a unified theoretical framework for reinforcement learning from human feedback using general $f$-divergences, proposing two algorithms with proven efficiency and performance bounds.

Contribution

It introduces a holistic analysis of $f$-divergence regularization in RLHF and presents two novel algorithms with theoretical guarantees.

Findings

Achieves $O(\log T)$ regret bounds.

Establishes $O(1/T)$ sub-optimality gap.

First to provide performance bounds for general $f$-divergence regularized online RLHF.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone technique for post-training large language models. While most existing approaches rely on the reverse KL-regularization, recent empirical studies have begun exploring alternative divergences (e.g., forward KL, chi-squared) as regularizers in RLHF. However, a unified theoretical understanding of general $f$-divergence regularization remains under-explored. To fill this gap, this work develops a comprehensive theoretical framework for online RLHF with a general $f$-divergence regularized objective. Rather than treating each possible divergence function individually, we adopt a holistic perspective across the entire function class and propose two algorithms based on distinct sampling principles. The first extends the classical optimism principle with a carefully designed exploration bonus, while the second introduces a new method that exploits the sensitivity of the optimal policy to reward perturbations under $f$-divergence regularization. Theoretical analysis shows that $O(\log T)$ regret and $O(1/T)$ sub-optimality gap are achievable, establishing provable efficiency of both algorithms and, to the best of our knowledge, the first performance bounds for online RLHF under general $f$-divergence regularization.