LMMCoDrive: Cooperative Driving with Large Multimodal Model

TL;DR

LMMCoDrive introduces a novel framework leveraging large multimodal models to improve decentralized cooperative scheduling and motion planning in autonomous urban vehicles, enhancing traffic efficiency and safety.

Contribution

The paper presents a new cooperative driving framework that integrates LMMs with decentralized optimization for autonomous vehicle coordination in urban environments.

Findings

Simulation shows improved traffic efficiency.

Enhanced safety through collision avoidance.

Effective decentralized optimization achieved.

Abstract

To address the intricate challenges of decentralized cooperative scheduling and motion planning in Autonomous Mobility-on-Demand (AMoD) systems, this paper introduces LMMCoDrive, a novel cooperative driving framework that leverages a Large Multimodal Model (LMM) to enhance traffic efficiency in dynamic urban environments. This framework seamlessly integrates scheduling and motion planning processes to ensure the effective operation of Cooperative Autonomous Vehicles (CAVs). The spatial relationship between CAVs and passenger requests is abstracted into a Bird's-Eye View (BEV) to fully exploit the potential of the LMM. Besides, trajectories are cautiously refined for each CAV while ensuring collision avoidance through safety constraints. A decentralized optimization strategy, facilitated by the Alternating Direction Method of Multipliers (ADMM) within the LMM framework, is proposed to drive the graph evolution of CAVs. Simulation results demonstrate the pivotal role and significant impact of LMM in optimizing CAV scheduling and enhancing decentralized cooperative optimization process for each vehicle. This marks a substantial stride towards achieving practical, efficient, and safe AMoD systems that are poised to revolutionize urban transportation. The code is available at https://github.com/henryhcliu/LMMCoDrive.