Boosting Sample Efficiency and Generalization in Multi-agent Reinforcement Learning via Equivariance

TL;DR

This paper introduces E2GN2, an equivariant graph neural network designed for multi-agent reinforcement learning, which significantly improves sample efficiency and generalization by leveraging symmetry structures, addressing exploration biases, and outperforming standard GNNs.

Contribution

The paper proposes E2GN2, a novel exploration-enhanced equivariant graph neural network that improves sample efficiency and generalization in multi-agent reinforcement learning.

Findings

E2GN2 outperforms standard GNNs in sample efficiency.

E2GN2 achieves greater final reward convergence.

E2GN2 shows 2x-5x better generalization in tests.

Abstract

Multi-Agent Reinforcement Learning (MARL) struggles with sample inefficiency and poor generalization [1]. These challenges are partially due to a lack of structure or inductive bias in the neural networks typically used in learning the policy. One such form of structure that is commonly observed in multi-agent scenarios is symmetry. The field of Geometric Deep Learning has developed Equivariant Graph Neural Networks (EGNN) that are equivariant (or symmetric) to rotations, translations, and reflections of nodes. Incorporating equivariance has been shown to improve learning efficiency and decrease error [ 2 ]. In this paper, we demonstrate that EGNNs improve the sample efficiency and generalization in MARL. However, we also show that a naive application of EGNNs to MARL results in poor early exploration due to a bias in the EGNN structure. To mitigate this bias, we present Exploration-enhanced Equivariant Graph Neural Networks or E2GN2. We compare E2GN2 to other common function approximators using common MARL benchmarks MPE and SMACv2. E2GN2 demonstrates a significant improvement in sample efficiency, greater final reward convergence, and a 2x-5x gain in over standard GNNs in our generalization tests. These results pave the way for more reliable and effective solutions in complex multi-agent systems.