Stochastic Prediction of Multi-Agent Interactions from Partial Observations

TL;DR

This paper introduces a Graph-VRNN model that combines learned dynamics and visual information to predict and understand multi-agent interactions from partial observations, demonstrating superior performance on sports datasets.

Contribution

The paper proposes a novel graph-structured variational recurrent neural network that integrates temporal and visual data for multi-agent interaction prediction.

Findings

Outperforms baselines on basketball trajectory data

Effective in predicting future states of multi-agent systems

Applicable to real and simulated sports environments

Abstract

We present a method that learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents. Our method is based on a graph-structured variational recurrent neural network (Graph-VRNN), which is trained end-to-end to infer the current state of the (partially observed) world, as well as to forecast future states. We show that our method outperforms various baselines on two sports datasets, one based on real basketball trajectories, and one generated by a soccer game engine.