Hierarchical Task Network Planning for Facilitating Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces SOMARL, a hierarchical framework combining HTN planning and MARL to improve cooperative multi-agent learning in sparse reward environments by leveraging symbolic knowledge and domain guidance.

Contribution

The paper presents a novel hierarchical framework integrating HTN planning with MARL, utilizing symbolic knowledge to enhance exploration and performance in cooperative multi-agent tasks.

Findings

SOMARL outperforms state-of-the-art MARL methods on benchmarks.

Hierarchical structure effectively guides exploration in sparse reward settings.

Symbolic knowledge integration improves learning efficiency and success rate.

Abstract

Exploring sparse reward multi-agent reinforcement learning (MARL) environments with traps in a collaborative manner is a complex task. Agents typically fail to reach the goal state and fall into traps, which affects the overall performance of the system. To overcome this issue, we present SOMARL, a framework that uses prior knowledge to reduce the exploration space and assist learning. In SOMARL, agents are treated as part of the MARL environment, and symbolic knowledge is embedded using a tree structure to build a knowledge hierarchy. The framework has a two-layer hierarchical structure, comprising a hybrid module with a Hierarchical Task Network (HTN) planning and meta-controller at the higher level, and a MARL-based interactive module at the lower level. The HTN module and meta-controller use Hierarchical Domain Definition Language (HDDL) and the option framework to formalize symbolic knowledge and obtain domain knowledge and a symbolic option set, respectively. Moreover, the HTN module leverages domain knowledge to guide low-level agent exploration by assisting the meta-controller in selecting symbolic options. The meta-controller further computes intrinsic rewards of symbolic options to limit exploration behavior and adjust HTN planning solutions as needed. We evaluate SOMARL on two benchmarks, FindTreasure and MoveBox, and report superior performance over state-of-the-art MARL and subgoal-based baselines for MARL environments significantly.