AccMER: Accelerating Multi-Agent Experience Replay with Cache Locality-aware Prioritization

TL;DR

AccMER enhances multi-agent reinforcement learning efficiency by reusing high-priority experiences to improve cache locality, reducing training time by over 25% without sacrificing reward performance.

Contribution

The paper introduces AccMER, a novel method that reuses high-priority experiences over multiple steps to improve cache efficiency in multi-agent experience replay.

Findings

Achieves 25.4% reduction in training time on Predator-Prey environment.

Maintains comparable mean reward performance to existing algorithms.

Effectively improves cache locality and reduces data movement in multi-agent RL.

Abstract

Multi-Agent Experience Replay (MER) is a key component of off-policy reinforcement learning~(RL) algorithms. By remembering and reusing experiences from the past, experience replay significantly improves the stability of RL algorithms and their learning efficiency. In many scenarios, multiple agents interact in a shared environment during online training under centralized training and decentralized execution~(CTDE) paradigm. Current multi-agent reinforcement learning~(MARL) algorithms consider experience replay with uniform sampling or based on priority weights to improve transition data sample efficiency in the sampling phase. However, moving transition data histories for each agent through the processor memory hierarchy is a performance limiter. Also, as the agents' transitions continuously renew every iteration, the finite cache capacity results in increased cache misses. To this end, we propose \name, that repeatedly reuses the transitions~(experiences) for a window of $n$ steps in order to improve the cache locality and minimize the transition data movement, instead of sampling new transitions at each step. Specifically, our optimization uses priority weights to select the transitions so that only high-priority transitions will be reused frequently, thereby improving the cache performance. Our experimental results on the Predator-Prey environment demonstrate the effectiveness of reusing the essential transitions based on the priority weights, where we observe an end-to-end training time reduction of $25.4\%$~(for $32$ agents) compared to existing prioritized MER algorithms without notable degradation in the mean reward.