Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactions

TL;DR

This paper introduces Role Play (RP), a novel framework for multi-agent reinforcement learning that uses role embeddings and role prediction to enhance adaptability and generalization in diverse, real-world multi-agent scenarios.

Contribution

RP leverages role embeddings and role prediction to improve policy diversity and adaptability, addressing limitations of self-play in complex multi-agent environments.

Findings

RP outperforms baselines in cooperative and mixed-motive games.

RP demonstrates robustness and adaptability with unseen agents.

Theoretical proof supports the effectiveness of role-based policies.

Abstract

Zero-shot coordination problem in multi-agent reinforcement learning (MARL), which requires agents to adapt to unseen agents, has attracted increasing attention. Traditional approaches often rely on the Self-Play (SP) framework to generate a diverse set of policies in a policy pool, which serves to improve the generalization capability of the final agent. However, these frameworks may struggle to capture the full spectrum of potential strategies, especially in real-world scenarios that demand agents balance cooperation with competition. In such settings, agents need strategies that can adapt to varying and often conflicting goals. Drawing inspiration from Social Value Orientation (SVO)-where individuals maintain stable value orientations during interactions with others-we propose a novel framework called \emph{Role Play} (RP). RP employs role embeddings to transform the challenge of policy diversity into a more manageable diversity of roles. It trains a common policy with role embedding observations and employs a role predictor to estimate the joint role embeddings of other agents, helping the learning agent adapt to its assigned role. We theoretically prove that an approximate optimal policy can be achieved by optimizing the expected cumulative reward relative to an approximate role-based policy. Experimental results in both cooperative (Overcooked) and mixed-motive games (Harvest, CleanUp) reveal that RP consistently outperforms strong baselines when interacting with unseen agents, highlighting its robustness and adaptability in complex environments.