Exploration by Random Distribution Distillation

TL;DR

This paper introduces Random Distribution Distillation (RDD), a novel exploration method in reinforcement learning that combines count-based and prediction-error approaches through stochastic target network outputs, enhancing exploration efficiency.

Contribution

RDD is a new exploration technique that models target network outputs as samples from a normal distribution, unifying count-based and prediction-error methods in RL.

Findings

RDD improves exploration in high-dimensional spaces.

Experimental results show RDD outperforms existing methods.

Theoretical analysis confirms RDD's effectiveness.

Abstract

Exploration remains a critical challenge in online reinforcement learning, as an agent must effectively explore unknown environments to achieve high returns. Currently, the main exploration algorithms are primarily count-based methods and curiosity-based methods, with prediction-error methods being a prominent example. In this paper, we propose a novel method called \textbf{R}andom \textbf{D}istribution \textbf{D}istillation (RDD), which samples the output of a target network from a normal distribution. RDD facilitates a more extensive exploration by explicitly treating the difference between the prediction network and the target network as an intrinsic reward. Furthermore, by introducing randomness into the output of the target network for a given state and modeling it as a sample from a normal distribution, intrinsic rewards are bounded by two key components: a pseudo-count term ensuring proper exploration decay and a discrepancy term accounting for predictor convergence. We demonstrate that RDD effectively unifies both count-based and prediction-error approaches. It retains the advantages of prediction-error methods in high-dimensional spaces, while also implementing an intrinsic reward decay mode akin to the pseudo-count method. In the experimental section, RDD is compared with more advanced methods in a series of environments. Both theoretical analysis and experimental results confirm the effectiveness of our approach in improving online exploration for reinforcement learning tasks.