Unsupervised Hierarchical Skill Discovery

TL;DR

This paper introduces an unsupervised method for hierarchical skill discovery in reinforcement learning that segments trajectories into reusable skills and constructs meaningful hierarchies without labels, improving structure and learning efficiency.

Contribution

It presents a novel grammar-based approach for unsupervised skill segmentation and hierarchy induction in high-dimensional environments, outperforming existing methods.

Findings

Produces more structured and meaningful hierarchies than baselines.

Accelerates and stabilizes learning in downstream tasks.

Effective in pixel-based environments like Craftax and Minecraft.

Abstract

We consider the problem of unsupervised skill segmentation and hierarchical structure discovery in reinforcement learning. While recent approaches have sought to segment trajectories into reusable skills or options, most rely on action labels, rewards, or handcrafted annotations, limiting their applicability. We propose a method that segments unlabelled trajectories into skills and induces a hierarchical structure over them using a grammar-based approach. The resulting hierarchy captures both low-level behaviours and their composition into higher-level skills. We evaluate our approach in high-dimensional, pixel-based environments, including Craftax and the full, unmodified version of Minecraft. Using metrics for skill segmentation, reuse, and hierarchy quality, we find that our method consistently produces more structured and semantically meaningful hierarchies than existing baselines. Furthermore, as a proof of concept for utility, we demonstrate that these discovered hierarchies accelerate and stabilise learning on downstream reinforcement learning tasks.