Structured Diversification Emergence via Reinforced Organization Control and Hierarchical Consensus Learning

TL;DR

This paper introduces Rochico, a novel multi-agent reinforcement learning framework that enhances cooperation and exploration through structured team formation and hierarchical consensus, outperforming current state-of-the-art methods.

Contribution

The paper proposes Rochico, a new MARL framework combining adaptive grouping, hierarchical consensus, and self-supervised rewards for improved cooperative behavior.

Findings

Outperforms SOTA algorithms in exploration efficiency.

Enhances cooperation strength in large-scale tasks.

Effectively promotes structured diversification emergence.

Abstract

When solving a complex task, humans will spontaneously form teams and to complete different parts of the whole task, respectively. Meanwhile, the cooperation between teammates will improve efficiency. However, for current cooperative MARL methods, the cooperation team is constructed through either heuristics or end-to-end blackbox optimization. In order to improve the efficiency of cooperation and exploration, we propose a structured diversification emergence MARL framework named {\sc{Rochico}} based on reinforced organization control and hierarchical consensus learning. {\sc{Rochico}} first learns an adaptive grouping policy through the organization control module, which is established by independent multi-agent reinforcement learning. Further, the hierarchical consensus module based on the hierarchical intentions with consensus constraint is introduced after team formation. Simultaneously, utilizing the hierarchical consensus module and a self-supervised intrinsic reward enhanced decision module, the proposed cooperative MARL algorithm {\sc{Rochico}} can output the final diversified multi-agent cooperative policy. All three modules are organically combined to promote the structured diversification emergence. Comparative experiments on four large-scale cooperation tasks show that {\sc{Rochico}} is significantly better than the current SOTA algorithms in terms of exploration efficiency and cooperation strength.