Large Language Models for Human-like Autonomous Driving: A Survey

TL;DR

This survey reviews how Large Language Models are being integrated into autonomous driving systems to enable more human-like behavior, discussing recent progress, challenges, and future research directions.

Contribution

It provides a comprehensive review of recent advances in applying LLMs to autonomous driving, highlighting key challenges and proposing future research avenues.

Findings

LLMs can enhance modular and end-to-end AD systems.

Integration challenges include real-time inference and safety.

Future directions involve addressing deployment costs and safety concerns.

Abstract

Large Language Models (LLMs), AI models trained on massive text corpora with remarkable language understanding and generation capabilities, are transforming the field of Autonomous Driving (AD). As AD systems evolve from rule-based and optimization-based methods to learning-based techniques like deep reinforcement learning, they are now poised to embrace a third and more advanced category: knowledge-based AD empowered by LLMs. This shift promises to bring AD closer to human-like AD. However, integrating LLMs into AD systems poses challenges in real-time inference, safety assurance, and deployment costs. This survey provides a comprehensive and critical review of recent progress in leveraging LLMs for AD, focusing on their applications in modular AD pipelines and end-to-end AD systems. We highlight key advancements, identify pressing challenges, and propose promising research directions to bridge the gap between LLMs and AD, thereby facilitating the development of more human-like AD systems. The survey first introduces LLMs' key features and common training schemes, then delves into their applications in modular AD pipelines and end-to-end AD, respectively, followed by discussions on open challenges and future directions. Through this in-depth analysis, we aim to provide insights and inspiration for researchers and practitioners working at the intersection of AI and autonomous vehicles, ultimately contributing to safer, smarter, and more human-centric AD technologies.