Inverse Reinforcement Learning in Swarm Systems

TL;DR

This paper extends inverse reinforcement learning to large-scale multi-agent swarm systems by introducing a new framework, reducing the problem to single-agent IRL, and demonstrating effective local reward modeling.

Contribution

It introduces the swarMDP framework, reduces multi-agent IRL to single-agent, and proposes a novel learning scheme tailored for swarm systems.

Findings

Successfully models local rewards from swarm data

Replicates observed global dynamics using learned rewards

Framework applicable to large-scale homogeneous systems

Abstract

Inverse reinforcement learning (IRL) has become a useful tool for learning behavioral models from demonstration data. However, IRL remains mostly unexplored for multi-agent systems. In this paper, we show how the principle of IRL can be extended to homogeneous large-scale problems, inspired by the collective swarming behavior of natural systems. In particular, we make the following contributions to the field: 1) We introduce the swarMDP framework, a sub-class of decentralized partially observable Markov decision processes endowed with a swarm characterization. 2) Exploiting the inherent homogeneity of this framework, we reduce the resulting multi-agent IRL problem to a single-agent one by proving that the agent-specific value functions in this model coincide. 3) To solve the corresponding control problem, we propose a novel heterogeneous learning scheme that is particularly tailored to the swarm setting. Results on two example systems demonstrate that our framework is able to produce meaningful local reward models from which we can replicate the observed global system dynamics.