Just Cluster It: An Approach for Exploration in High-Dimensions using Clustering and Pre-Trained Representations

TL;DR

This paper introduces a clustering-based exploration method in reinforcement learning that leverages pre-trained representations, demonstrating effectiveness in 3-D environments and surpassing existing methods in VizDoom and Habitat.

Contribution

It proposes a novel clustering approach using pre-trained and random representations for state counting, highlighting the benefits of pre-trained biases in exploration.

Findings

Random features can effectively cluster to count states in 3-D environments.

Pre-trained DINO representations outperform random features in complex visual environments.

The method surpasses other exploration techniques in VizDoom and Habitat environments.

Abstract

In this paper we adopt a representation-centric perspective on exploration in reinforcement learning, viewing exploration fundamentally as a density estimation problem. We investigate the effectiveness of clustering representations for exploration in 3-D environments, based on the observation that the importance of pixel changes between transitions is less pronounced in 3-D environments compared to 2-D environments, where pixel changes between transitions are typically distinct and significant. We propose a method that performs episodic and global clustering on random representations and on pre-trained DINO representations to count states, i.e, estimate pseudo-counts. Surprisingly, even random features can be clustered effectively to count states in 3-D environments, however when these become visually more complex, pre-trained DINO representations are more effective thanks to the pre-trained inductive biases in the representations. Overall, this presents a pathway for integrating pre-trained biases into exploration. We evaluate our approach on the VizDoom and Habitat environments, demonstrating that our method surpasses other well-known exploration methods in these settings.