PreND: Enhancing Intrinsic Motivation in Reinforcement Learning through Pre-trained Network Distillation

TL;DR

PreND introduces a novel method that leverages pre-trained models to improve intrinsic motivation in reinforcement learning, leading to better exploration and performance in sparse reward settings.

Contribution

The paper proposes PreND, a new approach that incorporates pre-trained representations into intrinsic motivation mechanisms, outperforming existing methods like RND in RL tasks.

Findings

PreND significantly outperforms RND in Atari experiments.

PreND provides more stable and meaningful intrinsic rewards.

Enhanced exploration and sample efficiency in sparse reward environments.

Abstract

Intrinsic motivation, inspired by the psychology of developmental learning in infants, stimulates exploration in agents without relying solely on sparse external rewards. Existing methods in reinforcement learning like Random Network Distillation (RND) face significant limitations, including (1) relying on raw visual inputs, leading to a lack of meaningful representations, (2) the inability to build a robust latent space, (3) poor target network initialization and (4) rapid degradation of intrinsic rewards. In this paper, we introduce Pre-trained Network Distillation (PreND), a novel approach to enhance intrinsic motivation in reinforcement learning (RL) by improving upon the widely used prediction-based method, RND. PreND addresses these challenges by incorporating pre-trained representation models into both the target and predictor networks, resulting in more meaningful and stable intrinsic rewards, while enhancing the representation learned by the model. We also tried simple but effective variants of the predictor network optimization by controlling the learning rate. Through experiments on the Atari domain, we demonstrate that PreND significantly outperforms RND, offering a more robust intrinsic motivation signal that leads to better exploration, improving overall performance and sample efficiency. This research highlights the importance of target and predictor networks representation in prediction-based intrinsic motivation, setting a new direction for improving RL agents' learning efficiency in sparse reward environments.