First Go, then Post-Explore: the Benefits of Post-Exploration in Intrinsic Motivation

TL;DR

This paper demonstrates that incorporating post-exploration into intrinsic motivation frameworks significantly improves the diversity of explored states and overall performance in reinforcement learning tasks, across various environments and settings.

Contribution

It provides the first ablation study isolating the effects of post-exploration within the IMGEP framework, showing its effectiveness and ease of integration.

Findings

Post-exploration increases state diversity in RL agents.

Post-exploration boosts performance in both discrete and continuous tasks.

The method is effective, method-agnostic, and easy to implement.

Abstract

Go-Explore achieved breakthrough performance on challenging reinforcement learning (RL) tasks with sparse rewards. The key insight of Go-Explore was that successful exploration requires an agent to first return to an interesting state ('Go'), and only then explore into unknown terrain ('Explore'). We refer to such exploration after a goal is reached as 'post-exploration'. In this paper, we present a clear ablation study of post-exploration in a general intrinsically motivated goal exploration process (IMGEP) framework, that the Go-Explore paper did not show. We study the isolated potential of post-exploration, by turning it on and off within the same algorithm under both tabular and deep RL settings on both discrete navigation and continuous control tasks. Experiments on a range of MiniGrid and Mujoco environments show that post-exploration indeed helps IMGEP agents reach more diverse states and boosts their performance. In short, our work suggests that RL researchers should consider to use post-exploration in IMGEP when possible since it is effective, method-agnostic and easy to implement.