Implicit Quantile Networks for Distributional Reinforcement Learning

TL;DR

This paper introduces Implicit Quantile Networks, a novel distributional reinforcement learning method that uses quantile regression to model return distributions, leading to improved Atari game performance and enabling risk-sensitive policies.

Contribution

It presents a flexible, state-of-the-art distributional RL approach using implicit quantile functions, advancing the modeling of return distributions and risk sensitivity.

Findings

Achieved superior performance on 57 Atari games.

Enabled analysis of risk-sensitive policies.

Demonstrated the effectiveness of implicit quantile modeling.

Abstract

In this work, we build on recent advances in distributional reinforcement learning to give a generally applicable, flexible, and state-of-the-art distributional variant of DQN. We achieve this by using quantile regression to approximate the full quantile function for the state-action return distribution. By reparameterizing a distribution over the sample space, this yields an implicitly defined return distribution and gives rise to a large class of risk-sensitive policies. We demonstrate improved performance on the 57 Atari 2600 games in the ALE, and use our algorithm's implicitly defined distributions to study the effects of risk-sensitive policies in Atari games.