ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Driving

TL;DR

ReasonPlan introduces a novel multimodal large language model framework for closed-loop autonomous driving, combining holistic reasoning, self-supervised scene prediction, and decision reasoning to improve interpretability and generalization over traditional methods.

Contribution

It presents a new fine-tuning framework for MLLMs in autonomous driving, with a dual mechanism for reasoning and a curated decision reasoning dataset, outperforming existing imitation learning approaches.

Findings

Outperforms mainstream imitation learning by 19% L2 and 16.1 score on Bench2Drive

Demonstrates strong zero-shot generalization on unseen benchmarks

Curates a large, diverse decision reasoning dataset for autonomous driving

Abstract

Due to the powerful vision-language reasoning and generalization abilities, multimodal large language models (MLLMs) have garnered significant attention in the field of end-to-end (E2E) autonomous driving. However, their application to closed-loop systems remains underexplored, and current MLLM-based methods have not shown clear superiority to mainstream E2E imitation learning approaches. In this work, we propose ReasonPlan, a novel MLLM fine-tuning framework designed for closed-loop driving through holistic reasoning with a self-supervised Next Scene Prediction task and supervised Decision Chain-of-Thought process. This dual mechanism encourages the model to align visual representations with actionable driving context, while promoting interpretable and causally grounded decision making. We curate a planning-oriented decision reasoning dataset, namely PDR, comprising 210k diverse and high-quality samples. Our method outperforms the mainstream E2E imitation learning method by a large margin of 19% L2 and 16.1 driving score on Bench2Drive benchmark. Furthermore, ReasonPlan demonstrates strong zero-shot generalization on unseen DOS benchmark, highlighting its adaptability in handling zero-shot corner cases. Code and dataset will be found in https://github.com/Liuxueyi/ReasonPlan.