LVLM-MPC Collaboration for Autonomous Driving: A Safety-Aware and Task-Scalable Control Architecture

TL;DR

This paper introduces a safety-aware control architecture that combines large vision-language models with model predictive control to enhance autonomous driving safety, task flexibility, and decision-making reliability.

Contribution

It presents a novel integration framework that leverages LVLMs for high-level planning and MPC for low-level safety and feasibility, addressing safety concerns in LVLM-driven autonomous driving.

Findings

Successfully handles highway driving scenarios in simulation

Ensures safety and task feasibility through MPC feedback

Maintains flexibility and adaptability of LVLMs

Abstract

This paper proposes a novel Large Vision-Language Model (LVLM) and Model Predictive Control (MPC) integration framework that delivers both task scalability and safety for Autonomous Driving (AD). LVLMs excel at high-level task planning across diverse driving scenarios. However, since these foundation models are not specifically designed for driving and their reasoning is not consistent with the feasibility of low-level motion planning, concerns remain regarding safety and smooth task switching. This paper integrates LVLMs with MPC Builder, which automatically generates MPCs on demand, based on symbolic task commands generated by the LVLM, while ensuring optimality and safety. The generated MPCs can strongly assist the execution or rejection of LVLM-driven task switching by providing feedback on the feasibility of the given tasks and generating task-switching-aware MPCs. Our approach provides a safe, flexible, and adaptable control framework, bridging the gap between cutting-edge foundation models and reliable vehicle operation. We demonstrate the effectiveness of our approach through a simulation experiment, showing that our system can safely and effectively handle highway driving while maintaining the flexibility and adaptability of LVLMs.