Poly-Autoregressive Prediction for Modeling Interactions

TL;DR

This paper introduces Poly-Autoregressive (PAR) modeling, a framework for predicting multi-agent behaviors by reasoning about their states over time, applicable across social, autonomous, and interaction scenarios, outperforming traditional autoregressive methods.

Contribution

The paper presents a novel Poly-Autoregressive framework that models multiple agents' interactions using sequence tokens, demonstrating its effectiveness across diverse multi-agent prediction tasks.

Findings

PAR outperforms autoregressive models in three different scenarios.

PAR can be applied with minimal data pre-processing.

The framework is versatile across social, autonomous, and interaction domains.

Abstract

We introduce a simple framework for predicting the behavior of an agent in multi-agent settings. In contrast to autoregressive (AR) tasks, such as language processing, our focus is on scenarios with multiple agents whose interactions are shaped by physical constraints and internal motivations. To this end, we propose Poly-Autoregressive (PAR) modeling, which forecasts an ego agent's future behavior by reasoning about the ego agent's state history and the past and current states of other interacting agents. At its core, PAR represents the behavior of all agents as a sequence of tokens, each representing an agent's state at a specific timestep. With minimal data pre-processing changes, we show that PAR can be applied to three different problems: human action forecasting in social situations, trajectory prediction for autonomous vehicles, and object pose forecasting during hand-object interaction. Using a small proof-of-concept transformer backbone, PAR outperforms AR across these three scenarios. The project website can be found at https://neerja.me/PAR/.