Characterizing Speed Performance of Multi-Agent Reinforcement Learning

TL;DR

This paper analyzes the speed performance of multi-agent reinforcement learning algorithms, emphasizing latency and throughput, and provides a systematic analysis of bottlenecks on CPU platforms to guide future acceleration efforts.

Contribution

It introduces a taxonomy for MARL algorithms from an acceleration perspective and systematically analyzes three state-of-the-art algorithms' performance bottlenecks.

Findings

Identified key bottlenecks in MARL algorithms on CPU platforms.

Highlighted the importance of latency-bounded throughput as a performance metric.

Suggested opportunities for parallelization and acceleration in MARL implementations.

Abstract

Multi-Agent Reinforcement Learning (MARL) has achieved significant success in large-scale AI systems and big-data applications such as smart grids, surveillance, etc. Existing advancements in MARL algorithms focus on improving the rewards obtained by introducing various mechanisms for inter-agent cooperation. However, these optimizations are usually compute- and memory-intensive, thus leading to suboptimal speed performance in end-to-end training time. In this work, we analyze the speed performance (i.e., latency-bounded throughput) as the key metric in MARL implementations. Specifically, we first introduce a taxonomy of MARL algorithms from an acceleration perspective categorized by (1) training scheme and (2) communication method. Using our taxonomy, we identify three state-of-the-art MARL algorithms - Multi-Agent Deep Deterministic Policy Gradient (MADDPG), Target-oriented Multi-agent Communication and Cooperation (ToM2C), and Networked Multi-Agent RL (NeurComm) - as target benchmark algorithms, and provide a systematic analysis of their performance bottlenecks on a homogeneous multi-core CPU platform. We justify the need for MARL latency-bounded throughput to be a key performance metric in future literature while also addressing opportunities for parallelization and acceleration.