Multi-layer Abstraction for Nested Generation of Options (MANGO) in Hierarchical Reinforcement Learning

TL;DR

MANGO is a hierarchical reinforcement learning framework that decomposes complex tasks into multiple abstraction layers with nested options, improving sample efficiency, generalization, and interpretability in sparse reward environments.

Contribution

The paper introduces MANGO, a novel multi-layer abstraction framework with nested options for hierarchical RL, enhancing efficiency and transparency over existing methods.

Findings

Significant improvements in sample efficiency in grid environments.

Enhanced generalization capabilities compared to standard RL.

Increased interpretability of agent decision-making across layers.

Abstract

This paper introduces MANGO (Multilayer Abstraction for Nested Generation of Options), a novel hierarchical reinforcement learning framework designed to address the challenges of long-term sparse reward environments. MANGO decomposes complex tasks into multiple layers of abstraction, where each layer defines an abstract state space and employs options to modularize trajectories into macro-actions. These options are nested across layers, allowing for efficient reuse of learned movements and improved sample efficiency. The framework introduces intra-layer policies that guide the agent's transitions within the abstract state space, and task actions that integrate task-specific components such as reward functions. Experiments conducted in procedurally-generated grid environments demonstrate substantial improvements in both sample efficiency and generalization capabilities compared to standard RL methods. MANGO also enhances interpretability by making the agent's decision-making process transparent across layers, which is particularly valuable in safety-critical and industrial applications. Future work will explore automated discovery of abstractions and abstract actions, adaptation to continuous or fuzzy environments, and more robust multi-layer training strategies.