CPIG: Leveraging Consistency Policy with Intention Guidance for Multi-agent Exploration

TL;DR

This paper introduces CPIG, a novel multi-agent reinforcement learning approach that combines a multimodal consistency policy with intention sharing to improve exploration and cooperation, especially in sparse-reward environments.

Contribution

The paper proposes a new method, CPIG, integrating a multimodal consistency policy and intention sharing to enhance exploration and cooperation in multi-agent RL.

Findings

Significantly outperforms SOTA in sparse-reward environments by 20%.

Achieves comparable performance to baselines in dense-reward settings.

Effectively promotes exploration and cooperation among agents.

Abstract

Efficient exploration is crucial in cooperative multi-agent reinforcement learning (MARL), especially in sparse-reward settings. However, due to the reliance on the unimodal policy, existing methods are prone to falling into the local optima, hindering the effective exploration of better policies. Furthermore, in sparse-reward settings, each agent tends to receive a scarce reward, which poses significant challenges to inter-agent cooperation. This not only increases the difficulty of policy learning but also degrades the overall performance of multi-agent tasks. To address these issues, we propose a Consistency Policy with Intention Guidance (CPIG), with two primary components: (a) introducing a multimodal policy to enhance the agent's exploration capability, and (b) sharing the intention among agents to foster agent cooperation. For component (a), CPIG incorporates a Consistency model as the policy, leveraging its multimodal nature and stochastic characteristics to facilitate exploration. Regarding component (b), we introduce an Intention Learner to deduce the intention on the global state from each agent's local observation. This intention then serves as a guidance for the Consistency Policy, promoting cooperation among agents. The proposed method is evaluated in multi-agent particle environments (MPE) and multi-agent MuJoCo (MAMuJoCo). Empirical results demonstrate that our method not only achieves comparable performance to various baselines in dense-reward environments but also significantly enhances performance in sparse-reward settings, outperforming state-of-the-art (SOTA) algorithms by 20%.