Maximum-Entropy Multi-Agent Dynamic Games: Forward and Inverse Solutions

TL;DR

This paper introduces the Entropic Cost Equilibrium (ECE), a new stochastic Nash equilibrium concept for multi-agent dynamic games, along with algorithms for forward and inverse problems, demonstrated in traffic scenarios.

Contribution

It defines ECE as a novel equilibrium concept for bounded rationality in multi-agent games and develops algorithms for computing and inferring agents' cost functions in continuous spaces.

Findings

The Riccati algorithm computes exact ECE policies in linear-quadratic cases.

Iterative methods find local ECE policies in nonlinear scenarios.

The inverse algorithm accurately infers agents' costs from noisy trajectory data.

Abstract

In this paper, we study the problem of multiple stochastic agents interacting in a dynamic game scenario with continuous state and action spaces. We define a new notion of stochastic Nash equilibrium for boundedly rational agents, which we call the Entropic Cost Equilibrium (ECE). We show that ECE is a natural extension to multiple agents of Maximum Entropy optimality for single agents. We solve both the "forward" and "inverse" problems for the multi-agent ECE game. For the forward problem, we provide a Riccati algorithm to compute closed-form ECE feedback policies for the agents, which are exact in the Linear-Quadratic-Gaussian case. We give an iterative variant to find locally ECE feedback policies for the nonlinear case. For the inverse problem, we present an algorithm to infer the cost functions of the multiple interacting agents given noisy, boundedly rational input and state trajectory examples from agents acting in an ECE. The effectiveness of our algorithms is demonstrated in a simulated multi-agent collision avoidance scenario, and with data from the INTERACTION traffic dataset. In both cases, we show that, by taking into account the agents' game theoretic interactions using our algorithm, a more accurate model of agents' costs can be learned, compared with standard inverse optimal control methods.