Closing the Loop: Motion Prediction Models beyond Open-Loop Benchmarks

TL;DR

This paper critically evaluates how state-of-the-art motion prediction models perform within autonomous driving systems, revealing that open-loop accuracy does not always translate to better closed-loop driving, and that smaller models can sometimes outperform larger ones.

Contribution

It systematically assesses the relationship between motion prediction accuracy and closed-loop driving performance, highlighting the importance of factors beyond open-loop metrics and exploring downsized models.

Findings

Higher open-loop accuracy does not guarantee better closed-loop behavior.

Temporal consistency and planner compatibility are crucial for effective integration.

Smaller models with fewer parameters can achieve comparable or better closed-loop performance.

Abstract

Fueled by motion prediction competitions and benchmarks, recent years have seen the emergence of increasingly large learning based prediction models, many with millions of parameters, focused on improving open-loop prediction accuracy by mere centimeters. However, these benchmarks fail to assess whether such improvements translate to better performance when integrated into an autonomous driving stack. In this work, we systematically evaluate the interplay between state-of-the-art motion predictors and motion planners. Our results show that higher open-loop accuracy does not always correlate with better closed-loop driving behavior and that other factors, such as temporal consistency of predictions and planner compatibility, also play a critical role. Furthermore, we investigate downsized variants of these models, and, surprisingly, find that in some cases models with up to 86% fewer parameters yield comparable or even superior closed-loop driving performance. Our code is available at https://github.com/aumovio/pred2plan.