End-to-end Autonomous Driving: Challenges and Frontiers

TL;DR

This survey reviews over 270 papers on end-to-end autonomous driving, highlighting recent advancements, challenges, and future directions in the field, emphasizing the integration of foundation models and large-scale datasets.

Contribution

It provides a comprehensive analysis of current research, challenges, and trends in end-to-end autonomous driving, including a curated repository of literature and open-source projects.

Findings

End-to-end systems benefit from joint perception and planning optimization.

Challenges include multi-modality, interpretability, and robustness.

Recent progress in foundation models enhances driving frameworks.

Abstract

The autonomous driving community has witnessed a rapid growth in approaches that embrace an end-to-end algorithm framework, utilizing raw sensor input to generate vehicle motion plans, instead of concentrating on individual tasks such as detection and motion prediction. End-to-end systems, in comparison to modular pipelines, benefit from joint feature optimization for perception and planning. This field has flourished due to the availability of large-scale datasets, closed-loop evaluation, and the increasing need for autonomous driving algorithms to perform effectively in challenging scenarios. In this survey, we provide a comprehensive analysis of more than 270 papers, covering the motivation, roadmap, methodology, challenges, and future trends in end-to-end autonomous driving. We delve into several critical challenges, including multi-modality, interpretability, causal confusion, robustness, and world models, amongst others. Additionally, we discuss current advancements in foundation models and visual pre-training, as well as how to incorporate these techniques within the end-to-end driving framework. we maintain an active repository that contains up-to-date literature and open-source projects at https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving.