Generative Adversarial Exploration for Reinforcement Learning

TL;DR

This paper introduces GAEX, a novel exploration method for reinforcement learning that uses a generative adversarial network to encourage agents to visit less familiar states, improving performance on complex exploration tasks.

Contribution

The paper presents the first use of GANs for RL exploration, integrating a discriminator and generator to effectively identify and encourage visiting novel states.

Findings

GAEX improves exploration in challenging RL environments.

GAEX outperforms baseline methods on Montezuma's Revenge and Super Mario Bros.

The approach is simple to implement and computationally efficient.

Abstract

Exploration is crucial for training the optimal reinforcement learning (RL) policy, where the key is to discriminate whether a state visiting is novel. Most previous work focuses on designing heuristic rules or distance metrics to check whether a state is novel without considering such a discrimination process that can be learned. In this paper, we propose a novel method called generative adversarial exploration (GAEX) to encourage exploration in RL via introducing an intrinsic reward output from a generative adversarial network, where the generator provides fake samples of states that help discriminator identify those less frequently visited states. Thus the agent is encouraged to visit those states which the discriminator is less confident to judge as visited. GAEX is easy to implement and of high training efficiency. In our experiments, we apply GAEX into DQN and the DQN-GAEX algorithm achieves convincing performance on challenging exploration problems, including the game Venture, Montezuma's Revenge and Super Mario Bros, without further fine-tuning on complicate learning algorithms. To our knowledge, this is the first work to employ GAN in RL exploration problems.