Valeo4Cast: A Modular Approach to End-to-End Forecasting

TL;DR

This paper introduces a modular approach to end-to-end motion forecasting in autonomous driving, which improves performance by separately training detection, tracking, and forecasting modules and then finetuning them together.

Contribution

The authors propose a modular training and finetuning strategy for motion forecasting that outperforms end-to-end trained models and achieves first place in a major benchmark.

Findings

Significant performance improvement over end-to-end models.

First place in the Argoverse 2 Forecasting Challenge.

Modular approach with finetuning enhances forecasting accuracy.

Abstract

Motion forecasting is crucial in autonomous driving systems to anticipate the future trajectories of surrounding agents such as pedestrians, vehicles, and traffic signals. In end-to-end forecasting, the model must jointly detect and track from sensor data (cameras or LiDARs) the past trajectories of the different elements of the scene and predict their future locations. We depart from the current trend of tackling this task via end-to-end training from perception to forecasting, and instead use a modular approach. We individually build and train detection, tracking and forecasting modules. We then only use consecutive finetuning steps to integrate the modules better and alleviate compounding errors. We conduct an in-depth study on the finetuning strategies and it reveals that our simple yet effective approach significantly improves performance on the end-to-end forecasting benchmark. Consequently, our solution ranks first in the Argoverse 2 End-to-end Forecasting Challenge, with 63.82 mAPf. We surpass forecasting results by +17.1 points over last year's winner and by +13.3 points over this year's runner-up. This remarkable performance in forecasting can be explained by our modular paradigm, which integrates finetuning strategies and significantly outperforms the end-to-end-trained counterparts. The code, model weights and results are made available https://github.com/valeoai/valeo4cast.