Imitation Learning of Factored Multi-agent Reactive Models

TL;DR

This paper introduces a deep generative imitation learning approach for multi-agent systems, modeling agents' policies directly from observational data with uncertainty estimates, demonstrated on fly interaction data.

Contribution

It applies variational recurrent neural networks to multi-agent imitation learning, enabling stochastic policy inference without reward functions and providing well-calibrated uncertainty estimates.

Findings

Outperforms existing deterministic policy models in predictive accuracy.

Provides well-calibrated uncertainty estimates for future trajectories.

Demonstrates effectiveness on biological fly interaction data.

Abstract

We apply recent advances in deep generative modeling to the task of imitation learning from biological agents. Specifically, we apply variations of the variational recurrent neural network model to a multi-agent setting where we learn policies of individual uncoordinated agents acting based on their perceptual inputs and their hidden belief state. We learn stochastic policies for these agents directly from observational data, without constructing a reward function. An inference network learned jointly with the policy allows for efficient inference over the agent's belief state given a sequence of its current perceptual inputs and the prior actions it performed, which lets us extrapolate observed sequences of behavior into the future while maintaining uncertainty estimates over future trajectories. We test our approach on a dataset of flies interacting in a 2D environment, where we demonstrate better predictive performance than existing approaches which learn deterministic policies with recurrent neural networks. We further show that the uncertainty estimates over future trajectories we obtain are well calibrated, which makes them useful for a variety of downstream processing tasks.