VisioPath: Vision-Language Enhanced Model Predictive Control for Safe Autonomous Navigation in Mixed Traffic

TL;DR

VisioPath integrates vision-language models with model predictive control to enhance safe autonomous navigation in dynamic traffic, utilizing rich perception and optimization techniques to outperform traditional methods.

Contribution

This work introduces a novel framework combining VLMs and MPC with a perception pipeline and safety verification for improved autonomous driving in complex environments.

Findings

Outperforms conventional MPC baselines in simulations

Effectively perceives surrounding vehicles using zero-shot VLMs

Ensures collision-free trajectories through safety verification

Abstract

In this paper, we introduce VisioPath, a novel framework combining vision-language models (VLMs) with model predictive control (MPC) to enable safe autonomous driving in dynamic traffic environments. The proposed approach leverages a bird's-eye view video processing pipeline and zero-shot VLM capabilities to obtain structured information about surrounding vehicles, including their positions, dimensions, and velocities. Using this rich perception output, we construct elliptical collision-avoidance potential fields around other traffic participants, which are seamlessly integrated into a finite-horizon optimal control problem for trajectory planning. The resulting trajectory optimization is solved via differential dynamic programming with an adaptive regularization scheme and is embedded in an event-triggered MPC loop. To ensure collision-free motion, a safety verification layer is incorporated in the framework that provides an assessment of potential unsafe trajectories. Extensive simulations in Simulation of Urban Mobility (SUMO) demonstrate that VisioPath outperforms conventional MPC baselines across multiple metrics. By combining modern AI-driven perception with the rigorous foundation of optimal control, VisioPath represents a significant step forward in safe trajectory planning for complex traffic systems.