High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning

TL;DR

This paper introduces QCoFr, a novel multi-agent reinforcement learning framework that models high-order interactions efficiently and interpretably, improving cooperation and performance without combinatorial complexity.

Contribution

QCoFr is a new value decomposition method that captures arbitrary-order agent interactions with linear complexity and enhances interpretability using a variational information bottleneck.

Findings

QCoFr outperforms existing methods in cooperative tasks.

It provides interpretable insights into agent interactions.

The approach maintains linear complexity regardless of interaction order.

Abstract

The ability to model interactions among agents is crucial for effective coordination and understanding their cooperation mechanisms in multi-agent reinforcement learning (MARL). However, previous efforts to model high-order interactions have been primarily hindered by the combinatorial explosion or the opaque nature of their black-box network structures. In this paper, we propose a novel value decomposition framework, called Continued Fraction Q-Learning (QCoFr), which can flexibly capture arbitrary-order agent interactions with only linear complexity $\mathcal{O}\left({n}\right)$ in the number of agents, thus avoiding the combinatorial explosion when modeling rich cooperation. Furthermore, we introduce the variational information bottleneck to extract latent information for estimating credits. This latent information helps agents filter out noisy interactions, thereby significantly enhancing both cooperation and interpretability. Extensive experiments demonstrate that QCoFr not only consistently achieves better performance but also provides interpretability that aligns with our theoretical analysis.