Subgoal-based Hierarchical Reinforcement Learning for Multi-Agent Collaboration

TL;DR

This paper introduces a hierarchical reinforcement learning framework that autonomously generates subgoals and adapts dynamically, significantly improving multi-agent collaboration efficiency and stability in complex environments.

Contribution

It presents a novel hierarchical architecture with dynamic goal generation and an enhanced credit assignment mechanism for better multi-agent coordination.

Findings

Improved convergence speed over traditional algorithms

Enhanced adaptability and sample efficiency in multi-agent tasks

Superior performance in complex environments

Abstract

Recent advancements in reinforcement learning have made significant impacts across various domains, yet they often struggle in complex multi-agent environments due to issues like algorithm instability, low sampling efficiency, and the challenges of exploration and dimensionality explosion. Hierarchical reinforcement learning (HRL) offers a structured approach to decompose complex tasks into simpler sub-tasks, which is promising for multi-agent settings. This paper advances the field by introducing a hierarchical architecture that autonomously generates effective subgoals without explicit constraints, enhancing both flexibility and stability in training. We propose a dynamic goal generation strategy that adapts based on environmental changes. This method significantly improves the adaptability and sample efficiency of the learning process. Furthermore, we address the critical issue of credit assignment in multi-agent systems by synergizing our hierarchical architecture with a modified QMIX network, thus improving overall strategy coordination and efficiency. Comparative experiments with mainstream reinforcement learning algorithms demonstrate the superior convergence speed and performance of our approach in both single-agent and multi-agent environments, confirming its effectiveness and flexibility in complex scenarios. Our code is open-sourced at: \url{https://github.com/SICC-Group/GMAH}.