Feudal Graph Reinforcement Learning

TL;DR

This paper introduces Feudal Graph Reinforcement Learning (FGRL), a hierarchical approach with pyramidal message passing to improve global coordination and high-level planning in graph-based RL tasks.

Contribution

FGRL combines hierarchical reinforcement learning with a pyramidal message-passing architecture to enhance decision-making and task decomposition in graph-based control problems.

Findings

FGRL outperforms relevant baselines on graph clustering and MuJoCo tasks.

The hierarchical message-passing scheme facilitates learning of hierarchical policies.

Analysis shows improved information flow and decision-making at different levels.

Abstract

Graph-based representations and message-passing modular policies constitute prominent approaches to tackling composable control problems in reinforcement learning (RL). However, as shown by recent graph deep learning literature, such local message-passing operators can create information bottlenecks and hinder global coordination. The issue becomes more serious in tasks requiring high-level planning. In this work, we propose a novel methodology, named Feudal Graph Reinforcement Learning (FGRL), that addresses such challenges by relying on hierarchical RL and a pyramidal message-passing architecture. In particular, FGRL defines a hierarchy of policies where high-level commands are propagated from the top of the hierarchy down through a layered graph structure. The bottom layers mimic the morphology of the physical system, while the upper layers correspond to higher-order sub-modules. The resulting agents are then characterized by a committee of policies where actions at a certain level set goals for the level below, thus implementing a hierarchical decision-making structure that can naturally implement task decomposition. We evaluate the proposed framework on a graph clustering problem and MuJoCo locomotion tasks; simulation results show that FGRL compares favorably against relevant baselines. Furthermore, an in-depth analysis of the command propagation mechanism provides evidence that the introduced message-passing scheme favors learning hierarchical decision-making policies.