Diversity-Driven Exploration Strategy for Deep Reinforcement Learning

TL;DR

This paper introduces a diversity-driven exploration strategy for deep reinforcement learning that improves exploration efficiency and prevents local optima trapping by incorporating a distance-based measure into the loss function, validated on Atari games.

Contribution

The paper proposes a simple yet effective diversity-driven exploration method that can be integrated with existing RL algorithms, enhancing exploration and performance.

Findings

Outperforms baseline methods on Atari 2600 in exploration efficiency

Enhances exploration behaviors by adding a distance measure to the loss function

Stabilizes learning with an adaptive scaling method

Abstract

Efficient exploration remains a challenging research problem in reinforcement learning, especially when an environment contains large state spaces, deceptive local optima, or sparse rewards. To tackle this problem, we present a diversity-driven approach for exploration, which can be easily combined with both off- and on-policy reinforcement learning algorithms. We show that by simply adding a distance measure to the loss function, the proposed methodology significantly enhances an agent's exploratory behaviors, and thus preventing the policy from being trapped in local optima. We further propose an adaptive scaling method for stabilizing the learning process. Our experimental results in Atari 2600 show that our method outperforms baseline approaches in several tasks in terms of mean scores and exploration efficiency.