Constrained Ensemble Exploration for Unsupervised Skill Discovery

TL;DR

This paper introduces a novel unsupervised reinforcement learning framework that uses an ensemble of skills with state-distribution constraints to promote diverse and well-explored behaviors, outperforming existing methods.

Contribution

It proposes a new ensemble-based skill discovery method with state-distribution constraints, enhancing exploration and diversity in unsupervised RL.

Findings

Learns well-explored ensemble skills

Achieves superior performance on downstream tasks

Provides theoretical analysis of state entropy and skill distributions

Abstract

Unsupervised Reinforcement Learning (RL) provides a promising paradigm for learning useful behaviors via reward-free per-training. Existing methods for unsupervised RL mainly conduct empowerment-driven skill discovery or entropy-based exploration. However, empowerment often leads to static skills, and pure exploration only maximizes the state coverage rather than learning useful behaviors. In this paper, we propose a novel unsupervised RL framework via an ensemble of skills, where each skill performs partition exploration based on the state prototypes. Thus, each skill can explore the clustered area locally, and the ensemble skills maximize the overall state coverage. We adopt state-distribution constraints for the skill occupancy and the desired cluster for learning distinguishable skills. Theoretical analysis is provided for the state entropy and the resulting skill distributions. Based on extensive experiments on several challenging tasks, we find our method learns well-explored ensemble skills and achieves superior performance in various downstream tasks compared to previous methods.