Applications of Large Scale Foundation Models for Autonomous Driving

TL;DR

This paper explores how large foundation models and LLMs can be integrated into autonomous driving systems to leverage human knowledge, reasoning, and commonsense, aiming to address long-tail AI challenges.

Contribution

It categorizes and analyzes various techniques of applying foundation models and LLMs in autonomous driving, including simulation, world modeling, data annotation, and planning.

Findings

Foundation models enhance autonomous driving capabilities.

LLMs facilitate improved reasoning and knowledge integration.

Potential to address long-tailed AI issues in autonomous systems.

Abstract

Since DARPA Grand Challenges (rural) in 2004/05 and Urban Challenges in 2007, autonomous driving has been the most active field of AI applications. Recently powered by large language models (LLMs), chat systems, such as chatGPT and PaLM, emerge and rapidly become a promising direction to achieve artificial general intelligence (AGI) in natural language processing (NLP). There comes a natural thinking that we could employ these abilities to reformulate autonomous driving. By combining LLM with foundation models, it is possible to utilize the human knowledge, commonsense and reasoning to rebuild autonomous driving systems from the current long-tailed AI dilemma. In this paper, we investigate the techniques of foundation models and LLMs applied for autonomous driving, categorized as simulation, world model, data annotation and planning or E2E solutions etc.