Comparison of Multi-agent and Single-agent Inverse Learning on a Simulated Soccer Example

TL;DR

This paper compares the effectiveness of Multi-agent IRL and single-agent IRL in a simulated soccer environment, extending Bayesian IRL to handle state-action rewards and demonstrating MIRL's superior performance.

Contribution

It extends Bayesian IRL to handle state-action rewards and empirically compares IRL and MIRL in a multi-agent game setting.

Findings

MIRL outperforms IRL in the soccer example

IRL struggles to capture equilibrium information

Extended Bayesian IRL to state-action rewards

Abstract

We compare the performance of Inverse Reinforcement Learning (IRL) with the relative new model of Multi-agent Inverse Reinforcement Learning (MIRL). Before comparing the methods, we extend a published Bayesian IRL approach that is only applicable to the case where the reward is only state dependent to a general one capable of tackling the case where the reward depends on both state and action. Comparison between IRL and MIRL is made in the context of an abstract soccer game, using both a game model in which the reward depends only on state and one in which it depends on both state and action. Results suggest that the IRL approach performs much worse than the MIRL approach. We speculate that the underperformance of IRL is because it fails to capture equilibrium information in the manner possible in MIRL.