CORD: Generalizable Cooperation via Role Diversity

TL;DR

CORD introduces a hierarchical multi-agent reinforcement learning framework that enhances generalization by promoting role diversity, enabling agents to adapt to unseen collaborators without prior knowledge.

Contribution

The paper proposes a novel hierarchical MARL method, CORD, that maximizes role entropy with constraints to improve generalization in cooperative tasks.

Findings

CORD outperforms baselines in generalization tests.

Role diversity improves cooperation among agents.

Ablation studies confirm the effectiveness of the constrained objective.

Abstract

Cooperative multi-agent reinforcement learning (MARL) aims to develop agents that can collaborate effectively. However, most cooperative MARL methods overfit training agents, making learned policies not generalize well to unseen collaborators, which is a critical issue for real-world deployment. Some methods attempt to address the generalization problem but require prior knowledge or predefined policies of new teammates, limiting real-world applications. To this end, we propose a hierarchical MARL approach to enable generalizable cooperation via role diversity, namely CORD. CORD's high-level controller assigns roles to low-level agents by maximizing the role entropy with constraints. We show this constrained objective can be decomposed into causal influence in role that enables reasonable role assignment, and role heterogeneity that yields coherent, non-redundant role clusters. Evaluated on a variety of cooperative multi-agent tasks, CORD achieves better performance than baselines, especially in generalization tests. Ablation studies further demonstrate the efficacy of the constrained objective in generalizable cooperation.