OBLR-PO: A Theoretical Framework for Stable Reinforcement Learning

TL;DR

This paper introduces a unified theoretical framework for reinforcement learning that characterizes policy-gradient estimators, enabling principled improvements in training stability and performance for large language models.

Contribution

It provides a systematic theoretical analysis of policy-gradient estimators, deriving variance expressions, convergence guarantees, and an adaptive learning-rate schedule, leading to the OBLR-PO algorithm.

Findings

OBLR-PO improves training stability and performance.

Theoretical analysis guides adaptive learning rates based on SNR.

Experimental results show consistent gains on large language models.

Abstract

Existing reinforcement learning (RL)-based post-training methods for large language models have advanced rapidly, yet their design has largely been guided by heuristics rather than systematic theoretical principles. This gap limits our understanding of the properties of the gradient estimators and the associated optimization algorithms, thereby constraining opportunities to improve training stability and overall performance. In this work, we provide a unified theoretical framework that characterizes the statistical properties of commonly used policy-gradient estimators under mild assumptions. Our analysis establishes unbiasedness, derives exact variance expressions, and yields an optimization-loss upper bound that enables principled reasoning about learning dynamics. Building on these results, we prove convergence guarantees and derive an adaptive learning-rate schedule governed by the signal-to-noise ratio (SNR) of gradients. We further show that the variance-optimal baseline is a gradient-weighted estimator, offering a new principle for variance reduction and naturally enhancing stability beyond existing methods. These insights motivate Optimal Baseline and Learning-Rate Policy Optimization (OBLR-PO), an algorithm that jointly adapts learning rates and baselines in a theoretically grounded manner. Experiments on Qwen3-4B-Base and Qwen3-8B-Base demonstrate consistent gains over existing policy optimization methods, validating that our theoretical contributions translate into practical improvements in large-scale post-training.