ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving

TL;DR

ReCogDrive introduces a reinforced cognitive framework that unifies understanding and planning in autonomous driving, leveraging hierarchical data pipelines and diffusion models to improve safety, efficiency, and scene comprehension.

Contribution

The paper presents a novel integrated framework combining hierarchical data processing and diffusion planning to enhance end-to-end autonomous driving performance.

Findings

Achieves state-of-the-art results on NAVSIM and Bench2Drive benchmarks.

Demonstrates improved safety and reduced collisions in diverse scenarios.

Shows enhanced scene understanding and driving stability.

Abstract

Recent studies have explored leveraging the world knowledge and cognitive capabilities of Vision-Language Models (VLMs) to address the long-tail problem in end-to-end autonomous driving. However, existing methods typically formulate trajectory planning as a language modeling task, where physical actions are output in the language space, potentially leading to issues such as format-violating outputs, infeasible actions, and slow inference speeds. In this paper, we propose ReCogDrive, a novel Reinforced Cognitive framework for end-to-end autonomous Driving, unifying driving understanding and planning by integrating an autoregressive model with a diffusion planner. First, to instill human driving cognition into the VLM, we introduce a hierarchical data pipeline that mimics the sequential cognitive process of human drivers through three stages: generation, refinement, and quality control. Building on this cognitive foundation, we then address the language-action mismatch by injecting the VLM's learned driving priors into a diffusion planner to efficiently generate continuous and stable trajectories. Furthermore, to enhance driving safety and reduce collisions, we introduce a Diffusion Group Relative Policy Optimization (DiffGRPO) stage, reinforcing the planner for enhanced safety and comfort. Extensive experiments on the NAVSIM and Bench2Drive benchmarks demonstrate that ReCogDrive achieves state-of-the-art performance. Additionally, qualitative results across diverse driving scenarios and DriveBench highlight the model's scene comprehension. All code, model weights, and datasets will be made publicly available to facilitate subsequent research.