Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning

TL;DR

This paper reveals that the success of distributional reinforcement learning stems from an intrinsic entropy regularization that captures return distribution uncertainty, leading to improved exploration and policy performance.

Contribution

It uncovers the theoretical basis of distributional RL's benefits as an entropy regularization derived from categorical distributional loss, enhancing understanding of its exploration advantages.

Findings

Distributional RL's superiority is linked to a distribution-matching entropy regularization.

The derived entropy regularization implicitly guides exploration by modeling environmental uncertainty.

Experiments demonstrate the effectiveness of uncertainty-aware regularization in improving RL performance.

Abstract

The remarkable empirical performance of distributional reinforcement learning (RL) has garnered increasing attention to understanding its theoretical advantages over classical RL. By decomposing the categorical distributional loss commonly employed in distributional RL, we find that the potential superiority of distributional RL can be attributed to a derived distribution-matching entropy regularization. This less-studied entropy regularization aims to capture additional knowledge of return distribution beyond only its expectation, contributing to an augmented reward signal in policy optimization. In contrast to the vanilla entropy regularization in MaxEnt RL, which explicitly encourages exploration by promoting diverse actions, the novel entropy regularization derived from categorical distributional loss implicitly updates policies to align the learned policy with (estimated) environmental uncertainty. Finally, extensive experiments verify the significance of this uncertainty-aware regularization from distributional RL on the empirical benefits over classical RL. Our study offers an innovative exploration perspective to explain the intrinsic benefits of distributional learning in RL.