DisTop: Discovering a Topological representation to learn diverse and rewarding skills

TL;DR

DisTop introduces a novel topological approach for deep reinforcement learning that enables the discovery of diverse, rewarding skills through an unsupervised, hierarchical, and environment-agnostic framework, improving exploration especially in sparse reward settings.

Contribution

DisTop presents a new method combining topological environment modeling with hierarchical skill discovery, advancing exploration and performance in DRL.

Findings

DisTop is effective across high-dimensional data, images, and proprioceptive inputs.

It achieves state-of-the-art results on MuJoCo benchmarks.

DisTop outperforms existing hierarchical RL methods in sparse reward scenarios.

Abstract

The optimal way for a deep reinforcement learning (DRL) agent to explore is to learn a set of skills that achieves a uniform distribution of states. Following this,we introduce DisTop, a new model that simultaneously learns diverse skills and focuses on improving rewarding skills. DisTop progressively builds a discrete topology of the environment using an unsupervised contrastive loss, a growing network and a goal-conditioned policy. Using this topology, a state-independent hierarchical policy can select where the agent has to keep discovering skills in the state space. In turn, the newly visited states allows an improved learnt representation and the learning loop continues. Our experiments emphasize that DisTop is agnostic to the ground state representation and that the agent can discover the topology of its environment whether the states are high-dimensional binary data, images, or proprioceptive inputs. We demonstrate that this paradigm is competitiveon MuJoCo benchmarks with state-of-the-art algorithms on both single-task dense rewards and diverse skill discovery. By combining these two aspects, we showthat DisTop achieves state-of-the-art performance in comparison with hierarchical reinforcement learning (HRL) when rewards are sparse. We believe DisTop opens new perspectives by showing that bottom-up skill discovery combined with representation learning can unlock the exploration challenge in DRL.