Evaluating and Improving Graph-based Explanation Methods for Multi-Agent Coordination

TL;DR

This paper evaluates the effectiveness of graph neural network explanation methods in multi-agent coordination tasks, proposing a regularization technique to improve explanation quality while maintaining task performance.

Contribution

It introduces an attention entropy regularization for GAT policies that enhances explanation clarity in multi-agent systems, supported by theoretical and empirical analysis.

Findings

Regularization improves explanation quality across tasks

Minimizing attention entropy increases subgraph disparity

Method maintains task performance while enhancing explainability

Abstract

Graph Neural Networks (GNNs), developed by the graph learning community, have been adopted and shown to be highly effective in multi-robot and multi-agent learning. Inspired by this successful cross-pollination, we investigate and characterize the suitability of existing GNN explanation methods for explaining multi-agent coordination. We find that these methods have the potential to identify the most-influential communication channels that impact the team's behavior. Informed by our initial analyses, we propose an attention entropy regularization term that renders GAT-based policies more amenable to existing graph-based explainers. Intuitively, minimizing attention entropy incentivizes agents to limit their attention to the most influential or impactful agents, thereby easing the challenge faced by the explainer. We theoretically ground this intuition by showing that minimizing attention entropy increases the disparity between the explainer-generated subgraph and its complement. Evaluations across three tasks and three team sizes i) provides insights into the effectiveness of existing explainers, and ii) demonstrates that our proposed regularization consistently improves explanation quality without sacrificing task performance.