Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning

TL;DR

This paper introduces SCIC, a novel multi-agent reinforcement learning algorithm that uses situation-dependent causal influence to improve cooperation and exploration among agents, leading to better performance on benchmarks.

Contribution

The paper proposes a new intrinsic reward mechanism based on causal influence and situation-dependent analysis to enhance multi-agent cooperation.

Findings

SCIC outperforms state-of-the-art methods on MARL benchmarks.

The causal influence-based intrinsic reward promotes effective exploration.

Agents learn to better coordinate through situation-aware causal influence detection.

Abstract

Learning to collaborate has witnessed significant progress in multi-agent reinforcement learning (MARL). However, promoting coordination among agents and enhancing exploration capabilities remain challenges. In multi-agent environments, interactions between agents are limited in specific situations. Effective collaboration between agents thus requires a nuanced understanding of when and how agents' actions influence others. To this end, in this paper, we propose a novel MARL algorithm named Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning (SCIC), which incorporates a novel Intrinsic reward mechanism based on a new cooperation criterion measured by situation-dependent causal influence among agents. Our approach aims to detect inter-agent causal influences in specific situations based on the criterion using causal intervention and conditional mutual information. This effectively assists agents in exploring states that can positively impact other agents, thus promoting cooperation between agents. The resulting update links coordinated exploration and intrinsic reward distribution, which enhance overall collaboration and performance. Experimental results on various MARL benchmarks demonstrate the superiority of our method compared to state-of-the-art approaches.