Concentration Network for Reinforcement Learning of Large-Scale Multi-Agent Systems

TL;DR

This paper introduces ConcNet, a novel concentration network that prioritizes and aggregates entity observations in large-scale multi-agent systems, improving scalability and performance in reinforcement learning tasks.

Contribution

ConcNet is a new concentration network that explicitly scores, ranks, and prunes observed entities based on motivational indices, enhancing efficiency in large-scale multi-agent reinforcement learning.

Findings

ConcNet outperforms existing methods on LMAS benchmarks.

It demonstrates excellent scalability and flexibility.

The concentration policy gradient effectively learns policies from scratch.

Abstract

When dealing with a series of imminent issues, humans can naturally concentrate on a subset of these concerning issues by prioritizing them according to their contributions to motivational indices, e.g., the probability of winning a game. This idea of concentration offers insights into reinforcement learning of sophisticated Large-scale Multi-Agent Systems (LMAS) participated by hundreds of agents. In such an LMAS, each agent receives a long series of entity observations at each step, which can overwhelm existing aggregation networks such as graph attention networks and cause inefficiency. In this paper, we propose a concentration network called ConcNet. First, ConcNet scores the observed entities considering several motivational indices, e.g., expected survival time and state value of the agents, and then ranks, prunes, and aggregates the encodings of observed entities to extract features. Second, distinct from the well-known attention mechanism, ConcNet has a unique motivational subnetwork to explicitly consider the motivational indices when scoring the observed entities. Furthermore, we present a concentration policy gradient architecture that can learn effective policies in LMAS from scratch. Extensive experiments demonstrate that the presented architecture has excellent scalability and flexibility, and significantly outperforms existing methods on LMAS benchmarks.