Coding for Distributed Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a coded distributed learning framework to mitigate straggler effects in multi-agent reinforcement learning, improving training speed while maintaining accuracy through various coding schemes.

Contribution

It proposes a novel coded distributed learning framework for MARL that effectively handles stragglers, with a specific implementation for MADDPG and evaluation of multiple coding schemes.

Findings

Coded framework speeds up MARL training in presence of stragglers

Different coding schemes are effective in distributed MARL

Simulations show promising performance in multi-robot tasks

Abstract

This paper aims to mitigate straggler effects in synchronous distributed learning for multi-agent reinforcement learning (MARL) problems. Stragglers arise frequently in a distributed learning system, due to the existence of various system disturbances such as slow-downs or failures of compute nodes and communication bottlenecks. To resolve this issue, we propose a coded distributed learning framework, which speeds up the training of MARL algorithms in the presence of stragglers, while maintaining the same accuracy as the centralized approach. As an illustration, a coded distributed version of the multi-agent deep deterministic policy gradient(MADDPG) algorithm is developed and evaluated. Different coding schemes, including maximum distance separable (MDS)code, random sparse code, replication-based code, and regular low density parity check (LDPC) code are also investigated. Simulations in several multi-robot problems demonstrate the promising performance of the proposed framework.