When to Go, and When to Explore: The Benefit of Post-Exploration in Intrinsic Motivation

TL;DR

This paper systematically studies post-exploration in reinforcement learning, demonstrating its significant benefits and proposing adaptive strategies to optimize when and how long to post-explore, leading to improved performance.

Contribution

It introduces a systematic analysis of post-exploration, including adaptive methods for deciding its timing and duration, enhancing RL exploration strategies.

Findings

Post-exploration boosts RL performance more than tuning regular exploration.

Adaptive post-exploration further improves results.

Post-exploration's impact surpasses traditional exploration parameter tuning.

Abstract

Go-Explore achieved breakthrough performance on challenging reinforcement learning (RL) tasks with sparse rewards. The key insight of Go-Explore was that successful exploration requires an agent to first return to an interesting state ('Go'), and only then explore into unknown terrain ('Explore'). We refer to such exploration after a goal is reached as 'post-exploration'. In this paper we present a systematic study of post-exploration, answering open questions that the Go-Explore paper did not answer yet. First, we study the isolated potential of post-exploration, by turning it on and off within the same algorithm. Subsequently, we introduce new methodology to adaptively decide when to post-explore and for how long to post-explore. Experiments on a range of MiniGrid environments show that post-exploration indeed boosts performance (with a bigger impact than tuning regular exploration parameters), and this effect is further enhanced by adaptively deciding when and for how long to post-explore. In short, our work identifies adaptive post-exploration as a promising direction for RL exploration research.