Cell-Free Latent Go-Explore

TL;DR

This paper introduces Latent Go-Explore, a generalized exploration method for reinforcement learning that leverages learned latent representations to improve robustness and performance in complex environments without domain-specific knowledge.

Contribution

The paper extends the Go-Explore paradigm by removing the need for domain knowledge and cell partitioning, enabling more flexible and effective exploration in RL environments.

Findings

LGE outperforms state-of-the-art algorithms in hard-exploration tasks

LGE is more robust than traditional Go-Explore

LGE is compatible with any latent representation learning strategy

Abstract

In this paper, we introduce Latent Go-Explore (LGE), a simple and general approach based on the Go-Explore paradigm for exploration in reinforcement learning (RL). Go-Explore was initially introduced with a strong domain knowledge constraint for partitioning the state space into cells. However, in most real-world scenarios, drawing domain knowledge from raw observations is complex and tedious. If the cell partitioning is not informative enough, Go-Explore can completely fail to explore the environment. We argue that the Go-Explore approach can be generalized to any environment without domain knowledge and without cells by exploiting a learned latent representation. Thus, we show that LGE can be flexibly combined with any strategy for learning a latent representation. Our results indicate that LGE, although simpler than Go-Explore, is more robust and outperforms state-of-the-art algorithms in terms of pure exploration on multiple hard-exploration environments including Montezuma's Revenge. The LGE implementation is available as open-source at https://github.com/qgallouedec/lge.