Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals

TL;DR

This paper introduces a hierarchical reinforcement learning approach that uses a diffusion model and Gaussian Process uncertainty to generate and select effective subgoals, improving sample efficiency and performance.

Contribution

It presents a novel method combining diffusion models with Gaussian Process priors for subgoal generation in HRL, addressing the challenge of evolving low-level policies.

Findings

Outperforms prior HRL methods in sample efficiency

Achieves higher performance on continuous control benchmarks

Effectively captures complex subgoal distributions with uncertainty quantification

Abstract

Hierarchical reinforcement learning (HRL) learns to make decisions on multiple levels of temporal abstraction. A key challenge in HRL is that the low-level policy changes over time, making it difficult for the high-level policy to generate effective subgoals. To address this issue, the high-level policy must capture a complex subgoal distribution while also accounting for uncertainty in its estimates. We propose an approach that trains a conditional diffusion model regularized by a Gaussian Process (GP) prior to generate a complex variety of subgoals while leveraging principled GP uncertainty quantification. Building on this framework, we develop a strategy that selects subgoals from both the diffusion policy and GP's predictive mean. Our approach outperforms prior HRL methods in both sample efficiency and performance on challenging continuous control benchmarks.