MART: MultiscAle Relational Transformer Networks for Multi-agent Trajectory Prediction

TL;DR

This paper introduces MART, a hypergraph transformer network that models individual and group behaviors for multi-agent trajectory prediction, achieving state-of-the-art results on multiple real-world datasets.

Contribution

The paper proposes a novel hypergraph transformer architecture with an encoder combining PRT and HRT, and an Adaptive Group Estimator for better group relation inference.

Findings

Achieves 3.9% improvement in ADE on NBA dataset

Achieves 11.8% improvement in FDE on NBA dataset

Outperforms existing methods on SDD and ETH-UCY datasets

Abstract

Multi-agent trajectory prediction is crucial to autonomous driving and understanding the surrounding environment. Learning-based approaches for multi-agent trajectory prediction, such as primarily relying on graph neural networks, graph transformers, and hypergraph neural networks, have demonstrated outstanding performance on real-world datasets in recent years. However, the hypergraph transformer-based method for trajectory prediction is yet to be explored. Therefore, we present a MultiscAle Relational Transformer (MART) network for multi-agent trajectory prediction. MART is a hypergraph transformer architecture to consider individual and group behaviors in transformer machinery. The core module of MART is the encoder, which comprises a Pair-wise Relational Transformer (PRT) and a Hyper Relational Transformer (HRT). The encoder extends the capabilities of a relational transformer by introducing HRT, which integrates hyperedge features into the transformer mechanism, promoting attention weights to focus on group-wise relations. In addition, we propose an Adaptive Group Estimator (AGE) designed to infer complex group relations in real-world environments. Extensive experiments on three real-world datasets (NBA, SDD, and ETH-UCY) demonstrate that our method achieves state-of-the-art performance, enhancing ADE/FDE by 3.9%/11.8% on the NBA dataset. Code is available at https://github.com/gist-ailab/MART.