Enhanced DACER Algorithm with High Diffusion Efficiency

TL;DR

DACERv2 enhances diffusion-based online reinforcement learning by introducing a Q-gradient guided denoising process and temporal weighting, significantly improving efficiency and performance with fewer diffusion steps.

Contribution

This paper introduces DACERv2, a novel method that improves diffusion policy efficiency in online RL through auxiliary Q-gradient guidance and temporal weighting mechanisms.

Findings

DACERv2 outperforms classical and diffusion-based algorithms on OpenAI Gym benchmarks.

It achieves higher performance with only five diffusion steps.

Demonstrates greater multimodality in control environments.

Abstract

Due to their expressive capacity, diffusion models have shown great promise in offline RL and imitation learning. Diffusion Actor-Critic with Entropy Regulator (DACER) extended this capability to online RL by using the reverse diffusion process as a policy approximator, achieving state-of-the-art performance. However, it still suffers from a core trade-off: more diffusion steps ensure high performance but reduce efficiency, while fewer steps degrade performance. This remains a major bottleneck for deploying diffusion policies in real-time online RL. To mitigate this, we propose DACERv2, which leverages a Q-gradient field objective with respect to action as an auxiliary optimization target to guide the denoising process at each diffusion step, thereby introducing intermediate supervisory signals that enhance the efficiency of single-step diffusion. Additionally, we observe that the independence of the Q-gradient field from the diffusion time step is inconsistent with the characteristics of the diffusion process. To address this issue, a temporal weighting mechanism is introduced, allowing the model to effectively eliminate large-scale noise during the early stages and refine its outputs in the later stages. Experimental results on OpenAI Gym benchmarks and multimodal tasks demonstrate that, compared with classical and diffusion-based online RL algorithms, DACERv2 achieves higher performance in most complex control environments with only five diffusion steps and shows greater multimodality.