Perception in Plan: Coupled Perception and Planning for End-to-End Autonomous Driving

TL;DR

This paper introduces VeteranAD, a coupled perception and planning framework for end-to-end autonomous driving that integrates perception into planning, enabling targeted perception and improved driving performance.

Contribution

It proposes a perception-in-plan framework with multi-mode anchored trajectories and autoregressive perception, advancing end-to-end autonomous driving methods.

Findings

Achieves state-of-the-art performance on NAVSIM and Bench2Drive datasets.

Enhances planning accuracy through targeted perception along planning priors.

Demonstrates improved reliability in autonomous driving behaviors.

Abstract

End-to-end autonomous driving has achieved remarkable advancements in recent years. Existing methods primarily follow a perception-planning paradigm, where perception and planning are executed sequentially within a fully differentiable framework for planning-oriented optimization. We further advance this paradigm through a perception-in-plan framework design, which integrates perception into the planning process. This design facilitates targeted perception guided by evolving planning objectives over time, ultimately enhancing planning performance. Building on this insight, we introduce VeteranAD, a coupled perception and planning framework for end-to-end autonomous driving. By incorporating multi-mode anchored trajectories as planning priors, the perception module is specifically designed to gather traffic elements along these trajectories, enabling comprehensive and targeted perception. Planning trajectories are then generated based on both the perception results and the planning priors. To make perception fully serve planning, we adopt an autoregressive strategy that progressively predicts future trajectories while focusing on relevant regions for targeted perception at each step. With this simple yet effective design, VeteranAD fully unleashes the potential of planning-oriented end-to-end methods, leading to more accurate and reliable driving behavior. Extensive experiments on the NAVSIM and Bench2Drive datasets demonstrate that our VeteranAD achieves state-of-the-art performance.