Hierarchical Reinforcement Learning with Deep Nested Agents

TL;DR

This paper introduces the Deep Nested Agent framework, enhancing deep hierarchical reinforcement learning by propagating information from main to nested agents, leading to improved efficiency and performance in complex environments like Minecraft.

Contribution

The paper proposes a novel Deep Nested Agent framework that improves information flow in hierarchical reinforcement learning, addressing inefficiencies in complex problem domains.

Findings

Outperforms non-hierarchical single agents in Minecraft scenarios

Demonstrates improved convergence times and policies

Effective in complex, multi-level environments

Abstract

Deep hierarchical reinforcement learning has gained a lot of attention in recent years due to its ability to produce state-of-the-art results in challenging environments where non-hierarchical frameworks fail to learn useful policies. However, as problem domains become more complex, deep hierarchical reinforcement learning can become inefficient, leading to longer convergence times and poor performance. We introduce the Deep Nested Agent framework, which is a variant of deep hierarchical reinforcement learning where information from the main agent is propagated to the low level $nested$ agent by incorporating this information into the nested agent's state. We demonstrate the effectiveness and performance of the Deep Nested Agent framework by applying it to three scenarios in Minecraft with comparisons to a deep non-hierarchical single agent framework, as well as, a deep hierarchical framework.