Understanding Action Effects through Instrumental Empowerment in Multi-Agent Reinforcement Learning

TL;DR

This paper introduces Intended Cooperation Values (ICVs), an information-theoretic approach to analyze agent influence on co-players in multi-agent reinforcement learning, improving understanding of individual contributions without explicit reward signals.

Contribution

The paper proposes ICVs, a novel method based on Shapley values, to quantify agent influence on others' policies solely through policy analysis, advancing explainability in MARL.

Findings

ICVs effectively identify beneficial agent behaviors in cooperative and competitive tasks.

The method reveals how agents influence each other's decision certainty and strategy diversity.

ICVs enhance understanding of cooperation dynamics and agent influence in MARL.

Abstract

To reliably deploy Multi-Agent Reinforcement Learning (MARL) systems, it is crucial to understand individual agent behaviors. While prior work typically evaluates overall team performance based on explicit reward signals, it is unclear how to infer agent contributions in the absence of any value feedback. In this work, we investigate whether meaningful insights into agent behaviors can be extracted solely by analyzing the policy distribution. Inspired by the phenomenon that intelligent agents tend to pursue convergent instrumental values, we introduce Intended Cooperation Values (ICVs), a method based on information-theoretic Shapley values for quantifying each agent's causal influence on their co-players' instrumental empowerment. Specifically, ICVs measure an agent's action effect on its teammates' policies by assessing their decision (un)certainty and preference alignment. By analyzing action effects on policies and value functions across cooperative and competitive MARL tasks, our method identifies which agent behaviors are beneficial to team success, either by fostering deterministic decisions or by preserving flexibility for future action choices, while also revealing the extent to which agents adopt similar or diverse strategies. Our proposed method offers novel insights into cooperation dynamics and enhances explainability in MARL systems.