Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value Functions

TL;DR

This paper introduces a graph-structured distributed reinforcement learning framework for large-scale multi-agent systems, improving scalability and efficiency by leveraging local value functions and graph couplings.

Contribution

It proposes a novel graph-based framework with two RL approaches that enhance scalability and reduce sample complexity in multi-agent reinforcement learning.

Findings

Significantly improved scalability over centralized methods

Effective reduction in sample complexity under certain graph conditions

Efficient approximate solutions for densely coupled graphs

Abstract

Achieving distributed reinforcement learning (RL) for large-scale cooperative multi-agent systems (MASs) is challenging because: (i) each agent has access to only limited information; (ii) issues on convergence or computational complexity emerge due to the curse of dimensionality. In this paper, we propose a general computationally efficient distributed framework for cooperative multi-agent reinforcement learning (MARL) by utilizing the structures of graphs involved in this problem. We introduce three coupling graphs describing three types of inter-agent couplings in MARL, namely, the state graph, the observation graph and the reward graph. By further considering a communication graph, we propose two distributed RL approaches based on local value-functions derived from the coupling graphs. The first approach is able to reduce sample complexity significantly under specific conditions on the aforementioned four graphs. The second approach provides an approximate solution and can be efficient even for problems with dense coupling graphs. Here there is a trade-off between minimizing the approximation error and reducing the computational complexity. Simulations show that our RL algorithms have a significantly improved scalability to large-scale MASs compared with centralized and consensus-based distributed RL algorithms.