Community-based Multi-Agent Reinforcement Learning with Transfer and Active Exploration

TL;DR

This paper introduces a novel community-based multi-agent reinforcement learning framework that captures overlapping community structures, enabling transfer learning and active exploration with theoretical convergence guarantees.

Contribution

It presents the first MARL framework integrating community structures, transferability, and active learning, supported by provable convergence guarantees.

Findings

Supports transfer learning via community membership estimation.

Enables active exploration by prioritizing uncertain communities.

Provides convergence guarantees under linear function approximation.

Abstract

We propose a new framework for multi-agent reinforcement learning (MARL), where the agents cooperate in a time-evolving network with latent community structures and mixed memberships. Unlike traditional neighbor-based or fixed interaction graphs, our community-based framework captures flexible and abstract coordination patterns by allowing each agent to belong to multiple overlapping communities. Each community maintains shared policy and value functions, which are aggregated by individual agents according to personalized membership weights. We also design actor-critic algorithms that exploit this structure: agents inherit community-level estimates for policy updates and value learning, enabling structured information sharing without requiring access to other agents' policies. Importantly, our approach supports both transfer learning by adapting to new agents or tasks via membership estimation, and active learning by prioritizing uncertain communities during exploration. Theoretically, we establish convergence guarantees under linear function approximation for both actor and critic updates. To our knowledge, this is the first MARL framework that integrates community structure, transferability, and active learning with provable guarantees.