Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexity

TL;DR

This paper introduces a novel bilevel reinforcement learning approach that leverages hyper-gradient computation without requiring the lower-level problem to be convex, enabling more flexible and efficient algorithms.

Contribution

It develops a first-order hyper-gradient method for bilevel RL that bypasses lower-level convexity assumptions, with both model-based and model-free algorithms and proven convergence rates.

Findings

Hyper-gradient effectively combines exploitation and exploration.

Algorithms achieve $O(psilon^{-1})$ convergence rate.

Stochastic algorithm extends applicability with complexity analysis.

Abstract

Bilevel reinforcement learning (RL), which features intertwined two-level problems, has attracted growing interest recently. The inherent non-convexity of the lower-level RL problem is, however, to be an impediment to developing bilevel optimization methods. By employing the fixed point equation associated with the regularized RL, we characterize the hyper-gradient via fully first-order information, thus circumventing the assumption of lower-level convexity. This, remarkably, distinguishes our development of hyper-gradient from the general AID-based bilevel frameworks since we take advantage of the specific structure of RL problems. Moreover, we design both model-based and model-free bilevel reinforcement learning algorithms, facilitated by access to the fully first-order hyper-gradient. Both algorithms enjoy the convergence rate $O(\epsilon^{-1})$. To extend the applicability, a stochastic version of the model-free algorithm is proposed, along with results on its iteration and sample complexity. In addition, numerical experiments demonstrate that the hyper-gradient indeed serves as an integration of exploitation and exploration.