From General Relation Patterns to Task-Specific Decision-Making in Continual Multi-Agent Coordination

TL;DR

This paper introduces RPG, a novel method that leverages relation patterns and task-specific decision-makers to improve continual multi-agent reinforcement learning, effectively preventing forgetting and enabling zero-shot generalization.

Contribution

The paper proposes RPG, a new approach that uses relation patterns and hypernetworks to enhance task adaptation and mitigate catastrophic forgetting in Co-MARL.

Findings

RPG outperforms existing methods on SMAC and LBF benchmarks.

RPG effectively prevents catastrophic forgetting in continual learning.

RPG achieves zero-shot generalization to unseen tasks.

Abstract

Continual Multi-Agent Reinforcement Learning (Co-MARL) requires agents to address catastrophic forgetting issues while learning new coordination policies with the dynamics team. In this paper, we delve into the core of Co-MARL, namely Relation Patterns, which refer to agents' general understanding of interactions. In addition to generality, relation patterns exhibit task-specificity when mapped to different action spaces. To this end, we propose a novel method called General Relation Patterns-Guided Task-Specific Decision-Maker (RPG). In RPG, agents extract relation patterns from dynamic observation spaces using a relation capturer. These task-agnostic relation patterns are then mapped to different action spaces via a task-specific decision-maker generated by a conditional hypernetwork. To combat forgetting, we further introduce regularization items on both the relation capturer and the conditional hypernetwork. Results on SMAC and LBF demonstrate that RPG effectively prevents catastrophic forgetting when learning new tasks and achieves zero-shot generalization to unseen tasks.