Hierarchical Empowerment: Towards Tractable Empowerment-Based Skill Learning

TL;DR

This paper presents Hierarchical Empowerment, a new framework that makes computing empowerment more tractable for skill learning by integrating hierarchical RL concepts, enabling agents to learn a broader repertoire of skills over longer time scales.

Contribution

The paper introduces a variational lower bound for mutual information and a hierarchical architecture to compute empowerment over extended time horizons.

Findings

Agents with four levels learned skills covering 100 times larger surface areas.

The framework successfully applied to simulated robotics tasks.

Enhanced skill diversity and scalability demonstrated in experiments.

Abstract

General purpose agents will require large repertoires of skills. Empowerment -- the maximum mutual information between skills and states -- provides a pathway for learning large collections of distinct skills, but mutual information is difficult to optimize. We introduce a new framework, Hierarchical Empowerment, that makes computing empowerment more tractable by integrating concepts from Goal-Conditioned Hierarchical Reinforcement Learning. Our framework makes two specific contributions. First, we introduce a new variational lower bound on mutual information that can be used to compute empowerment over short horizons. Second, we introduce a hierarchical architecture for computing empowerment over exponentially longer time scales. We verify the contributions of the framework in a series of simulated robotics tasks. In a popular ant navigation domain, our four level agents are able to learn skills that cover a surface area over two orders of magnitude larger than prior work.