Sample Complexity Analysis for Constrained Bilevel Reinforcement Learning

TL;DR

This paper provides the first theoretical analysis of sample complexity for constrained bilevel reinforcement learning, proposing an algorithm with specific iteration and sample complexity bounds, and handling non-smooth optimization via the Moreau envelope.

Contribution

It introduces the Constrained Bilevel Subgradient Optimization (CBSO) algorithm with theoretical guarantees, addressing non-smoothness and constraints in bilevel RL.

Findings

Iteration complexity of O(ε^{-2})

Sample complexity of ~O(ε^{-4})

First analysis of policy gradient RL with non-smooth objectives

Abstract

Several important problem settings within the literature of reinforcement learning (RL), such as meta-learning, hierarchical learning, and RL from human feedback (RL-HF), can be modelled as bilevel RL problems. A lot has been achieved in these domains empirically; however, the theoretical analysis of bilevel RL algorithms hasn't received a lot of attention. In this work, we analyse the sample complexity of a constrained bilevel RL algorithm, building on the progress in the unconstrained setting. We obtain an iteration complexity of $O(\epsilon^{-2})$ and sample complexity of $\tilde{O}(\epsilon^{-4})$ for our proposed algorithm, Constrained Bilevel Subgradient Optimization (CBSO). We use a penalty-based objective function to avoid the issue of primal-dual gap and hyper-gradient in the context of a constrained bilevel problem setting. The penalty-based formulation to handle constraints requires analysis of non-smooth optimization. We are the first ones to analyse the generally parameterized policy gradient-based RL algorithm with a non-smooth objective function using the Moreau envelope.