Branching Reinforcement Learning

TL;DR

This paper introduces a novel Branching Reinforcement Learning model that generates tree-structured trajectories, providing new algorithms and theoretical bounds for regret minimization and reward-free exploration in complex decision processes.

Contribution

The paper proposes a new branching RL framework, establishes fundamental Bellman equations and lemmas, and develops efficient algorithms with tight bounds for RM and RFE metrics.

Findings

Bound the total variance by O(H^2) despite exponential trajectories

Developed polynomial-time algorithms for regret minimization and exploration

Established nearly tight upper and lower bounds for the proposed methods

Abstract

In this paper, we propose a novel Branching Reinforcement Learning (Branching RL) model, and investigate both Regret Minimization (RM) and Reward-Free Exploration (RFE) metrics for this model. Unlike standard RL where the trajectory of each episode is a single $H$-step path, branching RL allows an agent to take multiple base actions in a state such that transitions branch out to multiple successor states correspondingly, and thus it generates a tree-structured trajectory. This model finds important applications in hierarchical recommendation systems and online advertising. For branching RL, we establish new Bellman equations and key lemmas, i.e., branching value difference lemma and branching law of total variance, and also bound the total variance by only $O(H^2)$ under an exponentially-large trajectory. For RM and RFE metrics, we propose computationally efficient algorithms BranchVI and BranchRFE, respectively, and derive nearly matching upper and lower bounds. Our results are only polynomial in problem parameters despite exponentially-large trajectories.