Quantum Hierarchical Reinforcement Learning via Variational Quantum Circuits

TL;DR

This paper introduces a hybrid hierarchical reinforcement learning agent using variational quantum circuits, demonstrating potential advantages in efficiency and performance over classical methods.

Contribution

It develops a quantum-enhanced hierarchical RL architecture with design principles, showing improved efficiency and insights into quantum component impacts.

Findings

Quantum feature extractors outperform classical baselines.

Hybrid agent saves up to 66% trainable parameters.

Quantum option-value estimation can degrade performance.

Abstract

Reinforcement learning is one of the most challenging learning paradigms where efficacy and efficiency gains are extremely valuable. Hierarchical reinforcement learning is a variant that leverages temporal abstraction to structure decision-making. While parametrized quantum computations have shown success in non-hierarchical reinforcement learning, whether these advantages adapt to hierarchical decision-making remains a critical open question. In this work, we develop a hybrid hierarchical agent based on the option-critic architecture. This hybrid agent substitutes classical components with variational quantum circuits for feature extractors, option-value functions, termination functions, and intra-option policies. Evaluated on standard benchmarking environments, results show that a hybrid agent utilizing a quantum feature extractor can outperform classical baselines while saving up to 66\% trainable parameters. We also identify an architectural bottleneck that quantum option-value estimation severely degrades performance. Further ablation studies reveal how architectural choices of the quantum circuits affect performance. Our work establishes design principles for parameter-efficient hybrid hierarchical agents.