Matching Multiple Experts: On the Exploitability of Multi-Agent Imitation Learning

TL;DR

This paper investigates the limits of multi-agent imitation learning in multi-player games, demonstrating fundamental hardness results and proposing conditions under which low-exploitability policies can be learned effectively.

Contribution

It provides the first formal hardness results for offline multi-agent imitation learning and introduces strategic dominance assumptions to achieve low exploitability.

Findings

Impossibility results for low-exploitability policies in general n-player games.

A bound on Nash imitation gap under dominant strategy equilibria.

Generalization of results using best-response continuity and regularization.

Abstract

Multi-agent imitation learning (MA-IL) aims to learn optimal policies from expert demonstrations of interactions in multi-agent interactive domains. Despite existing guarantees on the performance of the resulting learned policies, characterizations of how far the learned polices are from a Nash equilibrium are missing for offline MA-IL. In this paper, we demonstrate impossibility and hardness results of learning low-exploitable policies in general $n$-player Markov Games. We do so by providing examples where even exact measure matching fails, and demonstrating a new hardness result on characterizing the Nash gap given a fixed measure matching error. We then show how these challenges can be overcome using strategic dominance assumptions on the expert equilibrium. Specifically, for the case of dominant strategy expert equilibria, assuming Behavioral Cloning error $\epsilon_{\text{BC}}$, this provides a Nash imitation gap of $\mathcal{O}\left(n\epsilon_{\text{BC}}/(1-\gamma)^2\right)$ for a discount factor $\gamma$. We generalize this result with a new notion of best-response continuity, and argue that this is implicitly encouraged by standard regularization techniques.