Unsupervised Anomaly Detection in Multi-Agent Trajectory Prediction via Transformer-Based Models

TL;DR

This paper introduces an unsupervised Transformer-based framework for detecting complex, safety-critical anomalies in multi-agent driving scenarios, surpassing traditional metrics and identifying subtle risks missed by existing methods.

Contribution

It presents a novel unsupervised anomaly detection approach using multi-agent Transformers, with a dual evaluation scheme for stability and physical relevance, and demonstrates effectiveness on real driving data.

Findings

Max residual aggregator achieves best physical alignment.

Framework detects 388 anomalies missed by traditional metrics.

Identified anomalies cluster into four interpretable risk types.

Abstract

Identifying safety-critical scenarios is essential for autonomous driving, but the rarity of such events makes supervised labeling impractical. Traditional rule-based metrics like Time-to-Collision are too simplistic to capture complex interaction risks, and existing methods lack a systematic way to verify whether statistical anomalies truly reflect physical danger. To address this gap, we propose an unsupervised anomaly detection framework based on a multi-agent Transformer that models normal driving and measures deviations through prediction residuals. A dual evaluation scheme has been proposed to assess both detection stability and physical alignment: Stability is measured using standard ranking metrics in which Kendall Rank Correlation Coefficient captures rank agreement and Jaccard index captures the consistency of the top-K selected items; Physical alignment is assessed through correlations with established Surrogate Safety Measures (SSM). Experiments on the NGSIM dataset demonstrate our framework's effectiveness: We show that the maximum residual aggregator achieves the highest physical alignment while maintaining stability. Furthermore, our framework identifies 388 unique anomalies missed by Time-to-Collision and statistical baselines, capturing subtle multi-agent risks like reactive braking under lateral drift. The detected anomalies are further clustered into four interpretable risk types, offering actionable insights for simulation and testing.