D2C-HRHR: Discrete Actions with Double Distributional Critics for High-Risk-High-Return Tasks

TL;DR

This paper introduces a novel reinforcement learning framework for high-risk-high-return tasks that models multimodal action distributions using discretization and a dual-critic architecture, leading to improved performance in complex control tasks.

Contribution

It proposes a new RL approach that discretizes actions, uses entropy regularization, and employs dual critics to better handle multimodal and risky actions in high-dimensional spaces.

Findings

Outperforms baseline methods on locomotion and manipulation benchmarks

Explicit modeling of multimodality improves risk management in RL

Discretization enables effective approximation of complex action distributions

Abstract

Tasks involving high-risk-high-return (HRHR) actions, such as obstacle crossing, often exhibit multimodal action distributions and stochastic returns. Most reinforcement learning (RL) methods assume unimodal Gaussian policies and rely on scalar-valued critics, which limits their effectiveness in HRHR settings. We formally define HRHR tasks and theoretically show that Gaussian policies cannot guarantee convergence to the optimal solution. To address this, we propose a reinforcement learning framework that (i) discretizes continuous action spaces to approximate multimodal distributions, (ii) employs entropy-regularized exploration to improve coverage of risky but rewarding actions, and (iii) introduces a dual-critic architecture for more accurate discrete value distribution estimation. The framework scales to high-dimensional action spaces, supporting complex control domains. Experiments on locomotion and manipulation benchmarks with high risks of failure demonstrate that our method outperforms baselines, underscoring the importance of explicitly modeling multimodality and risk in RL.