A Cram\'er Distance perspective on Quantile Regression based Distributional Reinforcement Learning

TL;DR

This paper explores the use of the Cramér distance in quantile regression for distributional reinforcement learning, revealing theoretical connections to the Wasserstein distance and proposing an efficient computation method.

Contribution

It proves the equivalence of Cramér and Wasserstein projections and introduces a low complexity algorithm for Cramér distance computation in DRL.

Findings

Cramér distance projection coincides with 1-Wasserstein projection.

Squared Cramér and quantile regression losses have collinear gradients under non-crossing constraints.

Proposed algorithm efficiently computes the Cramér distance for DRL.

Abstract

Distributional reinforcement learning (DRL) extends the value-based approach by approximating the full distribution over future returns instead of the mean only, providing a richer signal that leads to improved performances. Quantile Regression (QR) based methods like QR-DQN project arbitrary distributions into a parametric subset of staircase distributions by minimizing the 1-Wasserstein distance. However, due to biases in the gradients, the quantile regression loss is used instead for training, guaranteeing the same minimizer and enjoying unbiased gradients. Non-crossing constraints on the quantiles have been shown to improve the performance of QR-DQN for uncertainty-based exploration strategies. The contribution of this work is in the setting of fixed quantile levels and is twofold. First, we prove that the Cram\'er distance yields a projection that coincides with the 1-Wasserstein one and that, under non-crossing constraints, the squared Cram\'er and the quantile regression losses yield collinear gradients, shedding light on the connection between these important elements of DRL. Second, we propose a low complexity algorithm to compute the Cram\'er distance.