Distributional Reinforcement Learning with Regularized Wasserstein Loss

TL;DR

This paper introduces SinkhornDRL, a distributional reinforcement learning method using regularized Wasserstein loss, which improves performance and interpretability over existing algorithms, especially in multi-dimensional reward scenarios.

Contribution

We propose SinkhornDRL, a novel distributional RL algorithm leveraging Sinkhorn divergence, with proven contraction properties and improved empirical performance.

Findings

Outperforms existing algorithms on Atari games

Shows robustness in multi-dimensional reward settings

Provides theoretical contraction guarantees

Abstract

The empirical success of distributional reinforcement learning (RL) highly relies on the choice of distribution divergence equipped with an appropriate distribution representation. In this paper, we propose \textit{Sinkhorn distributional RL (SinkhornDRL)}, which leverages Sinkhorn divergence, a regularized Wasserstein loss, to minimize the difference between current and target Bellman return distributions. Theoretically, we prove the contraction properties of SinkhornDRL, aligning with the interpolation nature of Sinkhorn divergence between Wasserstein distance and Maximum Mean Discrepancy (MMD). The introduced SinkhornDRL enriches the family of distributional RL algorithms, contributing to interpreting the algorithm behaviors compared with existing approaches by our investigation into their relationships. Empirically, we show that SinkhornDRL consistently outperforms or matches existing algorithms on the Atari games suite and particularly stands out in the multi-dimensional reward setting. \thanks{Code is available in \url{https://github.com/datake/SinkhornDistRL}.}.