Scalable Multiagent Reinforcement Learning with Collective Influence Estimation

TL;DR

This paper introduces a scalable multiagent reinforcement learning framework using Collective Influence Estimation Networks, enabling efficient coordination in large teams without extensive communication or network growth.

Contribution

It proposes a novel Collective Influence Estimation Network (CIEN) that models agent influence solely from local observations, enhancing scalability and robustness in multiagent systems.

Findings

Achieves stable coordination with limited communication.

Maintains scalability as team size increases.

Improves robustness and deployment feasibility on real robots.

Abstract

Multiagent reinforcement learning (MARL) has attracted considerable attention due to its potential in addressing complex cooperative tasks. However, existing MARL approaches often rely on frequent exchanges of action or state information among agents to achieve effective coordination, which is difficult to satisfy in practical robotic systems. A common solution is to introduce estimator networks to model the behaviors of other agents and predict their actions; nevertheless, such designs cause the size and computational cost of the estimator networks to grow rapidly with the number of agents, thereby limiting scalability in large-scale systems. To address these challenges, this paper proposes a multiagent learning framework augmented with a Collective Influence Estimation Network (CIEN). By explicitly modeling the collective influence of other agents on the task object, each agent can infer critical interaction information solely from its local observations and the task object's states, enabling efficient collaboration without explicit action information exchange. The proposed framework effectively avoids network expansion as the team size increases; moreover, new agents can be incorporated without modifying the network structures of existing agents, demonstrating strong scalability. Experimental results on multiagent cooperative tasks based on the Soft Actor-Critic (SAC) algorithm show that the proposed method achieves stable and efficient coordination under communication-limited environments. Furthermore, policies trained with collective influence modeling are deployed on a real robotic platform, where experimental results indicate significantly improved robustness and deployment feasibility, along with reduced dependence on communication infrastructure.