PlannerRFT: Reinforcing Diffusion Planners through Closed-Loop and Sample-Efficient Fine-Tuning

TL;DR

PlannerRFT introduces a sample-efficient reinforcement fine-tuning framework for diffusion-based autonomous driving planners, improving robustness and adaptability without changing the inference pipeline, supported by a fast simulator.

Contribution

It proposes PlannerRFT with dual-branch optimization and introduces nuMax, a faster simulator, enabling scalable, adaptive, and robust diffusion planner fine-tuning.

Findings

Achieves state-of-the-art performance in autonomous driving planning.

Develops nuMax, a simulator 10 times faster than nuPlan.

Demonstrates emergence of distinct behaviors during learning.

Abstract

Diffusion-based planners have emerged as a promising approach for human-like trajectory generation in autonomous driving. Recent works incorporate reinforcement fine-tuning to enhance the robustness of diffusion planners through reward-oriented optimization in a generation-evaluation loop. However, they struggle to generate multi-modal, scenario-adaptive trajectories, hindering the exploitation efficiency of informative rewards during fine-tuning. To resolve this, we propose PlannerRFT, a sample-efficient reinforcement fine-tuning framework for diffusion-based planners. PlannerRFT adopts a dual-branch optimization that simultaneously refines the trajectory distribution and adaptively guides the denoising process toward more promising exploration, without altering the original inference pipeline. To support parallel learning at scale, we develop nuMax, an optimized simulator that achieves 10 times faster rollout compared to native nuPlan. Extensive experiments shows that PlannerRFT yields state-of-the-art performance with distinct behaviors emerging during the learning process.