CorrectAD: A Self-Correcting Agentic System to Improve End-to-end Planning in Autonomous Driving

TL;DR

CorrectAD is a novel self-correcting system for autonomous driving that leverages diffusion-based video generation and structured 3D layouts to improve the robustness of end-to-end planning, significantly reducing failure cases.

Contribution

The paper introduces DriveSora, a new generative model for high-fidelity, spatiotemporally consistent data simulation conditioned on 3D layouts, integrated into a self-correcting framework for autonomous driving.

Findings

CorrectAD corrects 62.5% and 49.8% of failure cases on two datasets.

It reduces collision rates by 39% and 27%.

The system is end-to-end, model-agnostic, and improves various planners.

Abstract

End-to-end planning methods are the de facto standard of the current autonomous driving system, while the robustness of the data-driven approaches suffers due to the notorious long-tail problem (i.e., rare but safety-critical failure cases). In this work, we explore whether recent diffusion-based video generation methods (a.k.a. world models), paired with structured 3D layouts, can enable a fully automated pipeline to self-correct such failure cases. We first introduce an agent to simulate the role of product manager, dubbed PM-Agent, which formulates data requirements to collect data similar to the failure cases. Then, we use a generative model that can simulate both data collection and annotation. However, existing generative models struggle to generate high-fidelity data conditioned on 3D layouts. To address this, we propose DriveSora, which can generate spatiotemporally consistent videos aligned with the 3D annotations requested by PM-Agent. We integrate these components into our self-correcting agentic system, CorrectAD. Importantly, our pipeline is an end-to-end model-agnostic and can be applied to improve any end-to-end planner. Evaluated on both nuScenes and a more challenging in-house dataset across multiple end-to-end planners, CorrectAD corrects 62.5% and 49.8% of failure cases, reducing collision rates by 39% and 27%, respectively.