Convergence and stability of Q-learning in Hierarchical Reinforcement Learning

TL;DR

This paper provides a theoretical analysis of Feudal Q-learning in Hierarchical Reinforcement Learning, establishing conditions for convergence and stability, and linking it to game-theoretic concepts, supported by experiments.

Contribution

It introduces a convergence and stability analysis for Feudal Q-learning using stochastic approximation and ODE methods, connecting it to game theory.

Findings

Proves convergence and stability of Feudal Q-learning under certain conditions.

Shows that the algorithm's updates reach an equilibrium interpretable as a game.

Experimental results support the theoretical predictions.

Abstract

Hierarchical Reinforcement Learning promises, among other benefits, to efficiently capture and utilize the temporal structure of a decision-making problem and to enhance continual learning capabilities, but theoretical guarantees lag behind practice. In this paper, we propose a Feudal Q-learning scheme and investigate under which conditions its coupled updates converge and are stable. By leveraging the theory of Stochastic Approximation and the ODE method, we present a theorem stating the convergence and stability properties of Feudal Q-learning. This provides a principled convergence and stability analysis tailored to Feudal RL. Moreover, we show that the updates converge to a point that can be interpreted as an equilibrium of a suitably defined game, opening the door to game-theoretic approaches to Hierarchical RL. Lastly, experiments based on the Feudal Q-learning algorithm support the outcomes anticipated by theory.