Quantifying Agent Interaction in Multi-agent Reinforcement Learning for Cost-efficient Generalization

TL;DR

This paper introduces the Level of Influence (LoI) metric to quantify agent interactions in multi-agent reinforcement learning, showing that diverse co-player training improves generalization and that LoI-guided resource allocation enhances performance under budget constraints.

Contribution

We propose the LoI metric to measure agent interaction strength and develop a LoI-guided resource allocation method for better training efficiency in MARL.

Findings

Diverse co-player training improves generalization in MARL.

LoI effectively predicts scenario-specific generalization improvements.

LoI-guided resource allocation outperforms uniform strategies under limited budgets.

Abstract

Generalization poses a significant challenge in Multi-agent Reinforcement Learning (MARL). The extent to which an agent is influenced by unseen co-players depends on the agent's policy and the specific scenario. A quantitative examination of this relationship sheds light on effectively training agents for diverse scenarios. In this study, we present the Level of Influence (LoI), a metric quantifying the interaction intensity among agents within a given scenario and environment. We observe that, generally, a more diverse set of co-play agents during training enhances the generalization performance of the ego agent; however, this improvement varies across distinct scenarios and environments. LoI proves effective in predicting these improvement disparities within specific scenarios. Furthermore, we introduce a LoI-guided resource allocation method tailored to train a set of policies for diverse scenarios under a constrained budget. Our results demonstrate that strategic resource allocation based on LoI can achieve higher performance than uniform allocation under the same computation budget.