Causal Mean Field Multi-Agent Reinforcement Learning

TL;DR

This paper introduces CMFQ, a causality-aware algorithm that enhances the scalability of multi-agent reinforcement learning by modeling and quantifying essential interactions, demonstrating robustness in environments with many agents.

Contribution

The paper proposes a novel causal mean-field Q-learning framework that incorporates structural causal models to identify and leverage key interactions, improving scalability in multi-agent settings.

Findings

CMFQ outperforms existing methods in large-agent environments.

The approach maintains robustness despite nonstationary environments.

Effective in both cooperative and mixed game scenarios.

Abstract

Scalability remains a challenge in multi-agent reinforcement learning and is currently under active research. A framework named mean-field reinforcement learning (MFRL) could alleviate the scalability problem by employing the Mean Field Theory to turn a many-agent problem into a two-agent problem. However, this framework lacks the ability to identify essential interactions under nonstationary environments. Causality contains relatively invariant mechanisms behind interactions, though environments are nonstationary. Therefore, we propose an algorithm called causal mean-field Q-learning (CMFQ) to address the scalability problem. CMFQ is ever more robust toward the change of the number of agents though inheriting the compressed representation of MFRL's action-state space. Firstly, we model the causality behind the decision-making process of MFRL into a structural causal model (SCM). Then the essential degree of each interaction is quantified via intervening on the SCM. Furthermore, we design the causality-aware compact representation for behavioral information of agents as the weighted sum of all behavioral information according to their causal effects. We test CMFQ in a mixed cooperative-competitive game and a cooperative game. The result shows that our method has excellent scalability performance in both training in environments containing a large number of agents and testing in environments containing much more agents.