XLM for Autonomous Driving Systems: A Comprehensive Review

TL;DR

This paper reviews how multimodal large language models (XLMs) can enhance autonomous driving systems by integrating diverse sensory data for improved decision-making and control.

Contribution

It provides a comprehensive overview of XLM architectures, deployment approaches, challenges, and future directions in the context of autonomous driving.

Findings

XLMs can effectively combine multimodal data for autonomous driving.

Current approaches show promise but face deployment challenges.

Future research is needed to fully integrate XLMs into ADS.

Abstract

Large Language Models (LLMs) have showcased remarkable proficiency in various information-processing tasks. These tasks span from extracting data and summarizing literature to generating content, predictive modeling, decision-making, and system controls. Moreover, Vision Large Models (VLMs) and Multimodal LLMs (MLLMs), which represent the next generation of language models, a.k.a., XLMs, can combine and integrate many data modalities with the strength of language understanding, thus advancing several information-based systems, such as Autonomous Driving Systems (ADS). Indeed, by combining language communication with multimodal sensory inputs, e.g., panoramic images and LiDAR or radar data, accurate driving actions can be taken. In this context, we provide in this survey paper a comprehensive overview of the potential of XLMs towards achieving autonomous driving. Specifically, we review the relevant literature on ADS and XLMs, including their architectures, tools, and frameworks. Then, we detail the proposed approaches to deploy XLMs for autonomous driving solutions. Finally, we provide the related challenges to XLM deployment for ADS and point to future research directions aiming to enable XLM adoption in future ADS frameworks.