HyperMARL: Adaptive Hypernetworks for Multi-Agent RL

TL;DR

HyperMARL introduces an agent-conditioned hypernetwork approach that enhances adaptive cooperation in multi-agent reinforcement learning by reducing gradient interference and preserving behavioral diversity, outperforming standard parameter sharing methods.

Contribution

It presents a novel hypernetwork-based method that decouples agent-specific parameters from observations, improving adaptivity and diversity without added complexity.

Findings

Achieves competitive performance across 22 MARL scenarios with up to 30 agents.

Reduces policy gradient variance compared to traditional parameter sharing.

Maintains behavioral diversity comparable to non-parameter sharing methods.

Abstract

Adaptive cooperation in multi-agent reinforcement learning (MARL) requires policies to express homogeneous, specialised, or mixed behaviours, yet achieving this adaptivity remains a critical challenge. While parameter sharing (PS) is standard for efficient learning, it notoriously suppresses the behavioural diversity required for specialisation. This failure is largely due to cross-agent gradient interference, a problem we find is surprisingly exacerbated by the common practice of coupling agent IDs with observations. Existing remedies typically add complexity through altered objectives, manual preset diversity levels, or sequential updates -- raising a fundamental question: can shared policies adapt without these intricacies? We propose a solution built on a key insight: an agent-conditioned hypernetwork can generate agent-specific parameters and decouple observation- and agent-conditioned gradients, directly countering the interference from coupling agent IDs with observations. Our resulting method, HyperMARL, avoids the complexities of prior work and empirically reduces policy gradient variance. Across diverse MARL benchmarks (22 scenarios, up to 30 agents), HyperMARL achieves performance competitive with six key baselines while preserving behavioural diversity comparable to non-parameter sharing methods, establishing it as a versatile and principled approach for adaptive MARL. The code is publicly available at https://github.com/KaleabTessera/HyperMARL.