MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning

TL;DR

MAGIC is a multi-agent reinforcement learning framework that estimates multi-step causal effects between agents and uses advantage-based gating to improve coordination and performance.

Contribution

It introduces a novel causal influence estimation method with advantage gating, enhancing multi-agent coordination in complex environments.

Findings

MAGIC outperforms prior methods on MPE and StarCraft benchmarks.

Achieves 26.9% and 10.1% average performance improvements.

Uses counterfactual interventions and advantage gating for effective exploration.

Abstract

A key challenge in multi-agent reinforcement learning (MARL) lies in designing learning signals that effectively promote coordination among agents. Designing such signals requires estimating how one agent's current action affects its teammates over future interaction steps. To address this, we introduce Multi-step Advantage-Gated Interventional Causal MARL (MAGIC), a framework that estimates multi-step action effects between agents and selectively converts them into intrinsic rewards. MAGIC uses counterfactual action interventions to compare teammate futures under factual and counterfactual branches, and introduces a gate based on advantage to direct exploration toward beneficial behaviors aligned with the task goal. Experiments on Multi-Agent Particle Environments (MPE) and StarCraft micromanagement benchmarks (SMAC and SMACv2) show that MAGIC consistently outperforms leading prior methods, with average relative final performance improvements of 26.9% and 10.1%, respectively.