Sample-Efficient Hypergradient Estimation for Decentralized Bi-Level Reinforcement Learning

TL;DR

This paper introduces a sample-efficient method for hypergradient estimation in decentralized bi-level reinforcement learning, enabling effective optimization when the leader cannot intervene directly in the follower's process.

Contribution

It presents a novel hypergradient formulation using the Boltzmann covariance trick, allowing efficient, sample-based optimization in high-dimensional, decentralized RL settings.

Findings

Effective hypergradient estimation from interaction samples.

First method for hypergradient-based optimization in decentralized 2-player Markov games.

Demonstrated success in discrete and continuous state tasks.

Abstract

Many strategic decision-making problems, such as environment design for warehouse robots, can be naturally formulated as bi-level reinforcement learning (RL), where a leader agent optimizes its objective while a follower solves a Markov decision process (MDP) conditioned on the leader's decisions. In many situations, a fundamental challenge arises when the leader cannot intervene in the follower's optimization process; it can only observe the optimization outcome. We address this decentralized setting by deriving the hypergradient of the leader's objective, i.e., the gradient of the leader's strategy that accounts for changes in the follower's optimal policy. Unlike prior hypergradient-based methods that require extensive data for repeated state visits or rely on gradient estimators whose complexity can increase substantially with the high-dimensional leader's decision space, we leverage the Boltzmann covariance trick to derive an alternative hypergradient formulation. This enables efficient hypergradient estimation solely from interaction samples, even when the leader's decision space is high-dimensional. Additionally, to our knowledge, this is the first method that enables hypergradient-based optimization for 2-player Markov games in decentralized settings. Experiments highlight the impact of hypergradient updates and demonstrate our method's effectiveness in both discrete and continuous state tasks.