Distributed Multi-Agent Reinforcement Learning with One-hop Neighbors and Compute Straggler Mitigation

TL;DR

This paper introduces DARL1N, a scalable multi-agent reinforcement learning method that restricts communication to one-hop neighbors, combined with a coded distributed architecture to mitigate stragglers, significantly reducing training time and improving efficiency.

Contribution

The paper presents DARL1N, a novel off-policy actor-critic MARL algorithm with one-hop neighbor communication, and a coded distributed learning framework to handle stragglers, enhancing scalability and robustness.

Findings

DARL1N reduces training time significantly.

The method maintains policy quality with increasing agents.

Coded architecture improves resilience to stragglers.

Abstract

Most multi-agent reinforcement learning (MARL) methods are limited in the scale of problems they can handle. With increasing numbers of agents, the number of training iterations required to find the optimal behaviors increases exponentially due to the exponentially growing joint state and action spaces. This paper tackles this limitation by introducing a scalable MARL method called Distributed multi-Agent Reinforcement Learning with One-hop Neighbors (DARL1N). DARL1N is an off-policy actor-critic method that addresses the curse of dimensionality by restricting information exchanges among the agents to one-hop neighbors when representing value and policy functions. Each agent optimizes its value and policy functions over a one-hop neighborhood, significantly reducing the learning complexity, yet maintaining expressiveness by training with varying neighbor numbers and states. This structure allows us to formulate a distributed learning framework to further speed up the training procedure. Distributed computing systems, however, contain straggler compute nodes, which are slow or unresponsive due to communication bottlenecks, software or hardware problems. To mitigate the detrimental straggler effect, we introduce a novel coded distributed learning architecture, which leverages coding theory to improve the resilience of the learning system to stragglers. Comprehensive experiments show that DARL1N significantly reduces training time without sacrificing policy quality and is scalable as the number of agents increases. Moreover, the coded distributed learning architecture improves training efficiency in the presence of stragglers.