Hierarchical Strategies for Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a hierarchical multi-agent reinforcement learning framework that predicts agent behaviors and plans strategies, achieving state-of-the-art results on complex tasks like StarCraft II and Google Research Football.

Contribution

It proposes a novel two-level hierarchical architecture with a latent policy and information-theoretic objectives, enabling agents to learn and coordinate strategies effectively.

Findings

Achieved over 95% win rate on full Google Research Football game.

First MARL algorithm to solve all StarCraft II super hard scenarios.

Outperformed all existing methods on benchmark tasks.

Abstract

Adequate strategizing of agents behaviors is essential to solving cooperative MARL problems. One intuitively beneficial yet uncommon method in this domain is predicting agents future behaviors and planning accordingly. Leveraging this point, we propose a two-level hierarchical architecture that combines a novel information-theoretic objective with a trajectory prediction model to learn a strategy. To this end, we introduce a latent policy that learns two types of latent strategies: individual $z_A$, and relational $z_R$ using a modified Graph Attention Network module to extract interaction features. We encourage each agent to behave according to the strategy by conditioning its local $Q$ functions on $z_A$, and we further equip agents with a shared $Q$ function that conditions on $z_R$. Additionally, we introduce two regularizers to allow predicted trajectories to be accurate and rewarding. Empirical results on Google Research Football (GRF) and StarCraft (SC) II micromanagement tasks show that our method establishes a new state of the art being, to the best of our knowledge, the first MARL algorithm to solve all super hard SC II scenarios as well as the GRF full game with a win rate higher than $95\%$, thus outperforming all existing methods. Videos and brief overview of the methods and results are available at: https://sites.google.com/view/hier-strats-marl/home.