Scaling Goal-based Exploration via Pruning Proto-goals

TL;DR

This paper introduces a method for scalable goal-based exploration in reinforcement learning by pruning a large proto-goal space, balancing generality and tractability, and demonstrating its effectiveness in complex environments.

Contribution

It proposes a novel approach to refine a vast proto-goal space into a manageable, meaningful set for improved exploration in RL.

Findings

Effective goal-conditioned exploration in three challenging environments.

Successful pruning of proto-goal space enhances controllability and relevance.

Demonstrates scalability in large, complex domains.

Abstract

One of the gnarliest challenges in reinforcement learning (RL) is exploration that scales to vast domains, where novelty-, or coverage-seeking behaviour falls short. Goal-directed, purposeful behaviours are able to overcome this, but rely on a good goal space. The core challenge in goal discovery is finding the right balance between generality (not hand-crafted) and tractability (useful, not too many). Our approach explicitly seeks the middle ground, enabling the human designer to specify a vast but meaningful proto-goal space, and an autonomous discovery process to refine this to a narrower space of controllable, reachable, novel, and relevant goals. The effectiveness of goal-conditioned exploration with the latter is then demonstrated in three challenging environments.