Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Driving

TL;DR

This paper introduces DACER-F, a real-time generative policy method for autonomous driving that uses flow matching and Langevin dynamics to generate actions in a single inference step, significantly reducing latency.

Contribution

The paper proposes DACER-F, integrating flow matching with online RL and Langevin dynamics to enable fast, competitive action generation for autonomous driving.

Findings

Outperforms baselines in complex driving simulations.

Achieves high scores on DeepMind Control Suite.

Maintains ultra-low inference latency.

Abstract

Reinforcement learning (RL) is a fundamental methodology in autonomous driving systems, where generative policies exhibit considerable potential by leveraging their ability to model complex distributions to enhance exploration. However, their inherent high inference latency severely impedes their deployment in real-time decision-making and control. To address this issue, we propose diffusion actor-critic with entropy regulator via flow matching (DACER-F) by introducing flow matching into online RL, enabling the generation of competitive actions in a single inference step. By leveraging Langevin dynamics and gradients of the Q-function, DACER-F dynamically optimizes actions from experience replay toward a target distribution that balances high Q-value information with exploratory behavior. The flow policy is then trained to efficiently learn a mapping from a simple prior distribution to this dynamic target. In complex multi-lane and intersection simulations, DACER-F outperforms baselines diffusion actor-critic with entropy regulator (DACER) and distributional soft actor-critic (DSAC), while maintaining an ultra-low inference latency. DACER-F further demonstrates its scalability on standard RL benchmark DeepMind Control Suite (DMC), achieving a score of 775.8 in the humanoid-stand task and surpassing prior methods. Collectively, these results establish DACER-F as a high-performance and computationally efficient RL algorithm.