Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuning

TL;DR

This paper introduces Gen-Drive, a novel autonomous driving framework that uses a diffusion-based scene generator and a learned reward model, combined with reinforcement learning fine-tuning, to improve planning and decision-making in complex traffic scenarios.

Contribution

The paper proposes a generation-then-evaluation paradigm for autonomous driving, integrating diffusion models and reward modeling with RL fine-tuning for enhanced planning performance.

Findings

Generation-then-evaluation strategy outperforms other learning-based methods.

Fine-tuned generative policy significantly improves planning accuracy.

Reward model based on VLM-assisted preference data enhances evaluation quality.

Abstract

Autonomous driving necessitates the ability to reason about future interactions between traffic agents and to make informed evaluations for planning. This paper introduces the \textit{Gen-Drive} framework, which shifts from the traditional prediction and deterministic planning framework to a generation-then-evaluation planning paradigm. The framework employs a behavior diffusion model as a scene generator to produce diverse possible future scenarios, thereby enhancing the capability for joint interaction reasoning. To facilitate decision-making, we propose a scene evaluator (reward) model, trained with pairwise preference data collected through VLM assistance, thereby reducing human workload and enhancing scalability. Furthermore, we utilize an RL fine-tuning framework to improve the generation quality of the diffusion model, rendering it more effective for planning tasks. We conduct training and closed-loop planning tests on the nuPlan dataset, and the results demonstrate that employing such a generation-then-evaluation strategy outperforms other learning-based approaches. Additionally, the fine-tuned generative driving policy shows significant enhancements in planning performance. We further demonstrate that utilizing our learned reward model for evaluation or RL fine-tuning leads to better planning performance compared to relying on human-designed rewards. Project website: https://mczhi.github.io/GenDrive.