Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning

TL;DR

This paper revisits the master-slave architecture in multi-agent deep reinforcement learning, integrating decentralized and centralized perspectives into a hierarchical framework that improves performance in complex tasks.

Contribution

It introduces a hierarchical master-slave architecture combining decentralized control and centralized oversight with key components like composed actions, learnable communication, and independent reasoning.

Findings

Outperforms recent methods in synthetic experiments

Achieves superior results in StarCraft micromanagement tasks

Demonstrates the effectiveness of hierarchical structure in multi-agent RL

Abstract

Many tasks in artificial intelligence require the collaboration of multiple agents. We exam deep reinforcement learning for multi-agent domains. Recent research efforts often take the form of two seemingly conflicting perspectives, the decentralized perspective, where each agent is supposed to have its own controller; and the centralized perspective, where one assumes there is a larger model controlling all agents. In this regard, we revisit the idea of the master-slave architecture by incorporating both perspectives within one framework. Such a hierarchical structure naturally leverages advantages from one another. The idea of combining both perspectives is intuitive and can be well motivated from many real world systems, however, out of a variety of possible realizations, we highlights three key ingredients, i.e. composed action representation, learnable communication and independent reasoning. With network designs to facilitate these explicitly, our proposal consistently outperforms latest competing methods both in synthetic experiments and when applied to challenging StarCraft micromanagement tasks.