Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges

TL;DR

This paper reviews recent advances in foundation models for autonomous driving trajectory planning, highlighting their architectures, capabilities, limitations, and open challenges in the field.

Contribution

It provides a comprehensive taxonomy and critical evaluation of 37 recent foundation-model-based trajectory planning approaches in autonomous driving.

Findings

Foundation models enable direct trajectory inference from raw sensory data.

Multi-modal models incorporating natural language expand application scope.

Many approaches lack open-source code and datasets, limiting reproducibility.

Abstract

The emergence of multi-modal foundation models has markedly transformed the technology for autonomous driving, shifting away from conventional and mostly hand-crafted design choices towards unified, foundation-model-based approaches, capable of directly inferring motion trajectories from raw sensory inputs. This new class of methods can also incorporate natural language as an additional modality, with Vision-Language-Action (VLA) models serving as a representative example. In this review, we provide a comprehensive examination of such methods through a unifying taxonomy to critically evaluate their architectural design choices, methodological strengths, and their inherent capabilities and limitations. Our survey covers 37 recently proposed approaches that span the landscape of trajectory planning with foundation models. Furthermore, we assess these approaches with respect to the openness of their source code and datasets, offering valuable information to practitioners and researchers. We provide an accompanying webpage that catalogues the methods based on our taxonomy, available at: https://github.com/fiveai/FMs-for-driving-trajectories