Hierarchical Reinforcement Learning for Deep Goal Reasoning: An Expressiveness Analysis

TL;DR

This paper analyzes the expressiveness of hierarchical reinforcement learning architectures, demonstrating that recurrent hierarchical frameworks are more expressive than feedforward ones, supported by theoretical analysis and experimental validation.

Contribution

It introduces the recurrent hierarchical framework (RHF), generalizing existing architectures, and provides an expressiveness comparison with hierarchical DQN using formal grammar analysis.

Findings

RHF is more expressive than HF.

Experimental results support theoretical analysis.

Certain tasks cannot be solved by HF architectures.

Abstract

Hierarchical DQN (h-DQN) is a two-level architecture of feedforward neural networks where the meta level selects goals and the lower level takes actions to achieve the goals. We show tasks that cannot be solved by h-DQN, exemplifying the limitation of this type of hierarchical framework (HF). We describe the recurrent hierarchical framework (RHF), generalizing architectures that use a recurrent neural network at the meta level. We analyze the expressiveness of HF and RHF using context-sensitive grammars. We show that RHF is more expressive than HF. We perform experiments comparing an implementation of RHF with two HF baselines; the results corroborate our theoretical findings.