Oracles & Followers: Stackelberg Equilibria in Deep Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a flexible multi-agent reinforcement learning framework for computing Stackelberg equilibria, enabling novel algorithmic strategies like multitask and meta-RL techniques, and demonstrates improved sample efficiency through experimental evaluation.

Contribution

It presents a general framework for Stackelberg equilibrium search in multi-agent RL, unifies previous methods, and introduces new approaches leveraging contextual policies for follower convergence.

Findings

Improved sample efficiency over previous methods

Framework encompasses various existing approaches

Novel use of contextual policies for follower convergence

Abstract

Stackelberg equilibria arise naturally in a range of popular learning problems, such as in security games or indirect mechanism design, and have received increasing attention in the reinforcement learning literature. We present a general framework for implementing Stackelberg equilibria search as a multi-agent RL problem, allowing a wide range of algorithmic design choices. We discuss how previous approaches can be seen as specific instantiations of this framework. As a key insight, we note that the design space allows for approaches not previously seen in the literature, for instance by leveraging multitask and meta-RL techniques for follower convergence. We propose one such approach using contextual policies, and evaluate it experimentally on both standard and novel benchmark domains, showing greatly improved sample efficiency compared to previous approaches. Finally, we explore the effect of adopting algorithm designs outside the borders of our framework.