DSAC: Distributional Soft Actor-Critic for Risk-Sensitive Reinforcement Learning

TL;DR

DSAC is a novel reinforcement learning algorithm that integrates distributional reward modeling with entropy-driven exploration, improving performance in both risk-sensitive and risk-neutral continuous control tasks.

Contribution

Introduces DSAC, a unified framework combining distributional reward modeling and entropy maximization for risk-sensitive reinforcement learning.

Findings

Outperforms baseline algorithms on continuous control tasks

Effective in both risk-neutral and risk-sensitive scenarios

Enhances exploration through entropy balancing

Abstract

We present Distributional Soft Actor-Critic (DSAC), a distributional reinforcement learning (RL) algorithm that combines the strengths of distributional information of accumulated rewards and entropy-driven exploration from Soft Actor-Critic (SAC) algorithm. DSAC models the randomness in both action and rewards, surpassing baseline performances on various continuous control tasks. Unlike standard approaches that solely maximize expected rewards, we propose a unified framework for risk-sensitive learning, one that optimizes the risk-related objective while balancing entropy to encourage exploration. Extensive experiments demonstrate DSAC's effectiveness in enhancing agent performances for both risk-neutral and risk-sensitive control tasks.