Measuring What Matters: Scenario-Driven Evaluation for Trajectory Predictors in Autonomous Driving

TL;DR

This paper introduces a new evaluation pipeline for trajectory predictors in autonomous driving that considers both accuracy and diversity, providing a more meaningful assessment of their impact on vehicle safety and decision-making.

Contribution

It proposes a comprehensive, scenario-aware evaluation method that better correlates predictor performance with autonomous vehicle driving success.

Findings

The new pipeline outperforms traditional error-based metrics in reflecting driving performance.

It effectively balances accuracy and diversity based on scenario criticality.

Experiments demonstrate improved predictor selection for safer autonomous driving.

Abstract

Being able to anticipate the motion of surrounding agents is essential for the safe operation of autonomous driving systems in dynamic situations. While various methods have been proposed for trajectory prediction, the current evaluation practices still rely on error-based metrics (e.g., ADE, FDE), which reveal the accuracy from a post-hoc view but ignore the actual effect the predictor brings to the self-driving vehicles (SDVs), especially in complex interactive scenarios: a high-quality predictor not only chases accuracy, but should also captures all possible directions a neighbor agent might move, to support the SDVs' cautious decision-making. Given that the existing metrics hardly account for this standard, in our work, we propose a comprehensive pipeline that adaptively evaluates the predictor's performance by two dimensions: accuracy and diversity. Based on the criticality of the driving scenario, these two dimensions are dynamically combined and result in a final score for the predictor's performance. Extensive experiments on a closed-loop benchmark using real-world datasets show that our pipeline yields a more reasonable evaluation than traditional metrics by better reflecting the correlation of the predictors' evaluation with the autonomous vehicles' driving performance. This evaluation pipeline shows a robust way to select a predictor that potentially contributes most to the SDV's driving performance.