Heterogeneous Value Decomposition Policy Fusion for Multi-Agent Cooperation

TL;DR

This paper introduces Heterogeneous Policy Fusion (HPF), a novel method that adaptively combines multiple value decomposition approaches to enhance cooperative multi-agent reinforcement learning performance.

Contribution

The paper proposes HPF, a framework that integrates diverse VD methods with an adaptive policy selection mechanism and constraints to improve cooperation and optimization.

Findings

HPF outperforms baseline methods on cooperative tasks.

Adaptive policy fusion enhances learning efficiency.

Constraints mitigate misleading updates from exploration.

Abstract

Value decomposition (VD) has become one of the most prominent solutions in cooperative multi-agent reinforcement learning. Most existing methods generally explore how to factorize the joint value and minimize the discrepancies between agent observations and characteristics of environmental states. However, direct decomposition may result in limited representation or difficulty in optimization. Orthogonal to designing a new factorization scheme, in this paper, we propose Heterogeneous Policy Fusion (HPF) to integrate the strengths of various VD methods. We construct a composite policy set to select policies for interaction adaptively. Specifically, this adaptive mechanism allows agents' trajectories to benefit from diverse policy transitions while incorporating the advantages of each factorization method. Additionally, HPF introduces a constraint between these heterogeneous policies to rectify the misleading update caused by the unexpected exploratory or suboptimal non-cooperation. Experimental results on cooperative tasks show HPF's superior performance over multiple baselines, proving its effectiveness and ease of implementation.