Bounded Ratio Reinforcement Learning

TL;DR

This paper introduces Bounded Ratio Reinforcement Learning (BRRL), a new framework that unifies trust region methods and PPO, providing theoretical guarantees and improved empirical performance across various domains.

Contribution

The paper develops the BRRL framework with an analytical optimal solution, introduces Bounded Policy Optimization (BPO), and extends it to LLM fine-tuning, connecting trust region methods with the Cross-Entropy Method.

Findings

BPO outperforms PPO in stability and final performance across MuJoCo, Atari, and IsaacLab environments.

BRRL provides a theoretical foundation that explains PPO's success and guarantees monotonic improvement.

GBPO effectively fine-tunes LLMs, matching or surpassing existing methods.

Abstract

Proximal Policy Optimization (PPO) has become the predominant algorithm for on-policy reinforcement learning due to its scalability and empirical robustness across domains. However, there is a significant disconnect between the underlying foundations of trust region methods and the heuristic clipped objective used in PPO. In this paper, we bridge this gap by introducing the Bounded Ratio Reinforcement Learning (BRRL) framework. We formulate a novel regularized and constrained policy optimization problem and derive its analytical optimal solution. We prove that this solution ensures monotonic performance improvement. To handle parameterized policy classes, we develop a policy optimization algorithm called Bounded Policy Optimization (BPO) that minimizes an advantage-weighted divergence between the policy and the analytic optimal solution from BRRL. We further establish a lower bound on the expected performance of the resulting policy in terms of the BPO loss function. Notably, our framework also provides a new theoretical lens to interpret the success of the PPO loss, and connects trust region policy optimization and the Cross-Entropy Method (CEM). We additionally extend BPO to Group-relative BPO (GBPO) for LLM fine-tuning. Empirical evaluations of BPO across MuJoCo, Atari, and complex IsaacLab environments (e.g., Humanoid locomotion), and of GBPO for LLM fine-tuning tasks, demonstrate that BPO and GBPO generally match or outperform PPO and GRPO in stability and final performance.