Extending NGU to Multi-Agent RL: A Preliminary Study

TL;DR

This paper extends the NGU reinforcement learning algorithm to multi-agent environments, demonstrating that shared experience buffers and intrinsic exploration enhance performance and stability in multi-agent tasks.

Contribution

It introduces a multi-agent extension of NGU, evaluates design choices like shared buffers and novelty sharing, and provides insights into effective configurations for multi-agent RL.

Findings

Shared replay buffer improves performance and stability.

Sharing episodic novelty is effective at k=1 but degrades at higher k.

Heterogeneous beta values do not outperform a small common value.

Abstract

The Never Give Up (NGU) algorithm has proven effective in reinforcement learning tasks with sparse rewards by combining episodic novelty and intrinsic motivation. In this work, we extend NGU to multi-agent environments and evaluate its performance in the simple_tag environment from the PettingZoo suite. Compared to a multi-agent DQN baseline, NGU achieves moderately higher returns and more stable learning dynamics. We investigate three design choices: (1) shared replay buffer versus individual replay buffers, (2) sharing episodic novelty among agents using different k thresholds, and (3) using heterogeneous values of the beta parameter. Our results show that NGU with a shared replay buffer yields the best performance and stability, highlighting that the gains come from combining NGU intrinsic exploration with experience sharing. Novelty sharing performs comparably when k = 1 but degrades learning for larger values. Finally, heterogeneous beta values do not improve over a small common value. These findings suggest that NGU can be effectively applied in multi-agent settings when experiences are shared and intrinsic exploration signals are carefully tuned.