Stochastically Dominant Distributional Reinforcement Learning

TL;DR

This paper introduces a novel distributional reinforcement learning approach using second-order stochastic dominance to better manage uncertainty, with a particle-based algorithm demonstrating improved risk-performance trade-offs.

Contribution

It proposes a new SSD-based distributional RL method, mapping the problem to Wasserstein gradient flows and providing a convergent particle algorithm.

Findings

SSD policy balances uncertainty and performance better than other risk measures.

The particle algorithm is proven to be optimal and convergent.

Experiments validate the effectiveness of the proposed approach.

Abstract

We describe a new approach for managing aleatoric uncertainty in the Reinforcement Learning (RL) paradigm. Instead of selecting actions according to a single statistic, we propose a distributional method based on the second-order stochastic dominance (SSD) relation. This compares the inherent dispersion of random returns induced by actions, producing a more comprehensive and robust evaluation of the environment's uncertainty. The necessary conditions for SSD require estimators to predict accurate second moments. To accommodate this, we map the distributional RL problem to a Wasserstein gradient flow, treating the distributional Bellman residual as a potential energy functional. We propose a particle-based algorithm for which we prove optimality and convergence. Our experiments characterize the algorithm performance and demonstrate how uncertainty and performance are better balanced using an \textsc{ssd} policy than with other risk measures.