Provable Hierarchy-Based Meta-Reinforcement Learning

TL;DR

This paper introduces a provable method for hierarchy-based meta-reinforcement learning, enabling efficient learning of hierarchical structures during meta-training with theoretical guarantees, addressing limitations of prior heuristic approaches.

Contribution

It provides the first theoretical analysis of hierarchy learning in meta-RL, with guarantees for recovering natural hierarchies and bounds on downstream task performance.

Findings

Guarantees for hierarchy recovery under diversity conditions

Sample-efficient meta-training for hierarchical structures

Regret bounds for downstream task performance

Abstract

Hierarchical reinforcement learning (HRL) has seen widespread interest as an approach to tractable learning of complex modular behaviors. However, existing work either assume access to expert-constructed hierarchies, or use hierarchy-learning heuristics with no provable guarantees. To address this gap, we analyze HRL in the meta-RL setting, where a learner learns latent hierarchical structure during meta-training for use in a downstream task. We consider a tabular setting where natural hierarchical structure is embedded in the transition dynamics. Analogous to supervised meta-learning theory, we provide "diversity conditions" which, together with a tractable optimism-based algorithm, guarantee sample-efficient recovery of this natural hierarchy. Furthermore, we provide regret bounds on a learner using the recovered hierarchy to solve a meta-test task. Our bounds incorporate common notions in HRL literature such as temporal and state/action abstractions, suggesting that our setting and analysis capture important features of HRL in practice.