Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning

TL;DR

This paper introduces MANGER, a novel method that uses observation novelty to dynamically reuse data in multi-agent reinforcement learning, significantly improving efficiency and behavioral diversity in complex cooperative tasks.

Contribution

The paper presents MANGER, a new approach employing Random Network Distillation to enhance sample reuse and diversity in multi-agent reinforcement learning.

Findings

Improved sample efficiency in MARL tasks.

Enhanced exploration and behavioral diversity.

Superior performance in Google Research Football and StarCraft II tasks.

Abstract

Recently, deep Multi-Agent Reinforcement Learning (MARL) has demonstrated its potential to tackle complex cooperative tasks, pushing the boundaries of AI in collaborative environments. However, the efficiency of these systems is often compromised by inadequate sample utilization and a lack of diversity in learning strategies. To enhance MARL performance, we introduce a novel sample reuse approach that dynamically adjusts policy updates based on observation novelty. Specifically, we employ a Random Network Distillation (RND) network to gauge the novelty of each agent's current state, assigning additional sample update opportunities based on the uniqueness of the data. We name our method Multi-Agent Novelty-GuidEd sample Reuse (MANGER). This method increases sample efficiency and promotes exploration and diverse agent behaviors. Our evaluations confirm substantial improvements in MARL effectiveness in complex cooperative scenarios such as Google Research Football and super-hard StarCraft II micromanagement tasks.