Invariance to Quantile Selection in Distributional Continuous Control

TL;DR

This paper extends distributional reinforcement learning algorithms from discrete to continuous actions, demonstrating that their performance is invariant to the number and placement of distributional atoms in continuous control tasks.

Contribution

It introduces a transfer of three distributional algorithms to continuous control and shows their invariance to atom configuration in this setting.

Findings

Distributional algorithms perform consistently across different atom configurations.

Performance invariance observed in continuous control tasks.

Empirical validation on PyBullet environments.

Abstract

In recent years distributional reinforcement learning has produced many state of the art results. Increasingly sample efficient Distributional algorithms for the discrete action domain have been developed over time that vary primarily in the way they parameterize their approximations of value distributions, and how they quantify the differences between those distributions. In this work we transfer three of the most well-known and successful of those algorithms (QR-DQN, IQN and FQF) to the continuous action domain by extending two powerful actor-critic algorithms (TD3 and SAC) with distributional critics. We investigate whether the relative performance of the methods for the discrete action space translates to the continuous case. To that end we compare them empirically on the pybullet implementations of a set of continuous control tasks. Our results indicate qualitative invariance regarding the number and placement of distributional atoms in the deterministic, continuous action setting.