Super Agents and Confounders: Influence of surrounding agents on vehicle trajectory prediction

TL;DR

This paper analyzes how surrounding agents influence vehicle trajectory prediction, revealing flaws in current models and proposing a Conditional Information Bottleneck to improve robustness and accuracy.

Contribution

It provides a comprehensive analysis of state-of-the-art predictors, identifies their instability, and introduces a CIB method to enhance prediction robustness without extra supervision.

Findings

Many surrounding agents degrade prediction accuracy.

Models learn unstable, non-causal decision schemes.

CIB improves prediction performance and robustness.

Abstract

In highly interactive driving scenes, trajectory prediction is conditioned on information from surrounding traffic participants such as cars and pedestrians. Our main contribution is a comprehensive analysis of state-of-the-art trajectory predictors, which reveals a surprising and critical flaw: many surrounding agents degrade prediction accuracy rather than improve it. Using Shapley-based attribution, we rigorously demonstrate that models learn unstable and non-causal decision-making schemes that vary significantly across training runs. Building on these insights, we propose to integrate a Conditional Information Bottleneck (CIB), which does not require additional supervision and is trained to effectively compress agent features as well as ignore those that are not beneficial for the prediction task. Comprehensive experiments using multiple datasets and model architectures demonstrate that this simple yet effective approach not only improves overall trajectory prediction performance in many cases but also increases robustness to different perturbations. Our results highlight the importance of selectively integrating contextual information, which can often contain spurious or misleading signals, in trajectory prediction. Moreover, we provide interpretable metrics for identifying non-robust behavior and present a promising avenue towards a solution.