DriveMRP: Enhancing Vision-Language Models with Synthetic Motion Data for Motion Risk Prediction

TL;DR

This paper introduces DriveMRP, a synthetic high-risk motion data generation method and a VLM-agnostic framework that significantly improves motion risk prediction accuracy in autonomous driving scenarios, demonstrating strong generalization in real-world tests.

Contribution

It presents a novel BEV-based motion simulation technique and a VLM-agnostic risk estimation framework that enhances vision-language models' ability to predict motion risks.

Findings

Accident recognition accuracy increased from 27.13% to 88.03% after fine-tuning.

Zero-shot evaluation accuracy improved from 29.42% to 68.50%.

Synthetic high-risk motion data effectively enhances model performance.

Abstract

Autonomous driving has seen significant progress, driven by extensive real-world data. However, in long-tail scenarios, accurately predicting the safety of the ego vehicle's future motion remains a major challenge due to uncertainties in dynamic environments and limitations in data coverage. In this work, we aim to explore whether it is possible to enhance the motion risk prediction capabilities of Vision-Language Models (VLM) by synthesizing high-risk motion data. Specifically, we introduce a Bird's-Eye View (BEV) based motion simulation method to model risks from three aspects: the ego-vehicle, other vehicles, and the environment. This allows us to synthesize plug-and-play, high-risk motion data suitable for VLM training, which we call DriveMRP-10K. Furthermore, we design a VLM-agnostic motion risk estimation framework, named DriveMRP-Agent. This framework incorporates a novel information injection strategy for global context, ego-vehicle perspective, and trajectory projection, enabling VLMs to effectively reason about the spatial relationships between motion waypoints and the environment. Extensive experiments demonstrate that by fine-tuning with DriveMRP-10K, our DriveMRP-Agent framework can significantly improve the motion risk prediction performance of multiple VLM baselines, with the accident recognition accuracy soaring from 27.13% to 88.03%. Moreover, when tested via zero-shot evaluation on an in-house real-world high-risk motion dataset, DriveMRP-Agent achieves a significant performance leap, boosting the accuracy from base_model's 29.42% to 68.50%, which showcases the strong generalization capabilities of our method in real-world scenarios.