Distributional Soft Actor-Critic with Diffusion Policy

TL;DR

This paper introduces DSAC-D, a distributional reinforcement learning algorithm utilizing diffusion models to accurately learn multimodal policies and distributions, significantly reducing bias and improving performance in complex control tasks.

Contribution

The paper presents a novel diffusion-based distributional RL framework that effectively models multimodal policies and reduces value estimation bias, achieving state-of-the-art results.

Findings

Achieves over 10% improvement in average return on control tasks.

Successfully characterizes multimodal distributions of driving styles.

Demonstrates state-of-the-art performance in 9 control tasks.

Abstract

Reinforcement learning has been proven to be highly effective in handling complex control tasks. Traditional methods typically use unimodal distributions, such as Gaussian distributions, to model the output of value distributions. However, unimodal distribution often and easily causes bias in value function estimation, leading to poor algorithm performance. This paper proposes a distributional reinforcement learning algorithm called DSAC-D (Distributed Soft Actor Critic with Diffusion Policy) to address the challenges of estimating bias in value functions and obtaining multimodal policy representations. A multimodal distributional policy iteration framework that can converge to the optimal policy was established by introducing policy entropy and value distribution function. A diffusion value network that can accurately characterize the distribution of multi peaks was constructed by generating a set of reward samples through reverse sampling using a diffusion model. Based on this, a distributional reinforcement learning algorithm with dual diffusion of the value network and the policy network was derived. MuJoCo testing tasks demonstrate that the proposed algorithm not only learns multimodal policy, but also achieves state-of-the-art (SOTA) performance in all 9 control tasks, with significant suppression of estimation bias and total average return improvement of over 10% compared to existing mainstream algorithms. The results of real vehicle testing show that DSAC-D can accurately characterize the multimodal distribution of different driving styles, and the diffusion policy network can characterize multimodal trajectories.