Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flows

TL;DR

This paper introduces NF-HIQL, a novel hierarchical reinforcement learning framework using normalizing flows to improve data efficiency and policy expressivity in complex, long-horizon tasks, with strong theoretical guarantees and empirical performance.

Contribution

It proposes a new flow-based hierarchical implicit Q-learning framework that enhances policy expressivity and data efficiency, with theoretical analysis and empirical validation.

Findings

NF-HIQL outperforms prior methods in diverse tasks.

It demonstrates improved robustness under limited data.

The framework provides theoretical guarantees on stability and generalization.

Abstract

Hierarchical goal-conditioned reinforcement learning (H-GCRL) provides a powerful framework for tackling complex, long-horizon tasks by decomposing them into structured subgoals. However, its practical adoption is hindered by poor data efficiency and limited policy expressivity, especially in offline or data-scarce regimes. In this work, Normalizing flow-based hierarchical implicit Q-learning (NF-HIQL), a novel framework that replaces unimodal gaussian policies with expressive normalizing flow policies at both the high- and low-levels of the hierarchy is introduced. This design enables tractable log-likelihood computation, efficient sampling, and the ability to model rich multimodal behaviors. New theoretical guarantees are derived, including explicit KL-divergence bounds for Real-valued non-volume preserving (RealNVP) policies and PAC-style sample efficiency results, showing that NF-HIQL preserves stability while improving generalization. Empirically, NF-HIQL is evaluted across diverse long-horizon tasks in locomotion, ball-dribbling, and multi-step manipulation from OGBench. NF-HIQL consistently outperforms prior goal-conditioned and hierarchical baselines, demonstrating superior robustness under limited data and highlighting the potential of flow-based architectures for scalable, data-efficient hierarchical reinforcement learning.