Opponent-aware Role-based Learning in Team Competitive Markov Games

TL;DR

This paper introduces RAC, a novel multi-agent reinforcement learning method that learns dynamic, diverse roles within teams and predicts opponent roles, leading to higher rewards in competitive settings.

Contribution

RAC is the first approach to learn emergent, opponent-aware roles in team Markov games using an actor-critic framework with role encoding.

Findings

RAC outperforms state-of-the-art baselines in reward metrics.

Agents learn diverse and opponent-aware policies.

Method demonstrates effectiveness in two competitive games.

Abstract

Team competition in multi-agent Markov games is an increasingly important setting for multi-agent reinforcement learning, due to its general applicability in modeling many real-life situations. Multi-agent actor-critic methods are the most suitable class of techniques for learning optimal policies in the team competition setting, due to their flexibility in learning agent-specific critic functions, which can also learn from other agents. In many real-world team competitive scenarios, the roles of the agents naturally emerge, in order to aid in coordination and collaboration within members of the teams. However, existing methods for learning emergent roles rely heavily on the Q-learning setup which does not allow learning of agent-specific Q-functions. In this paper, we propose RAC, a novel technique for learning the emergent roles of agents within a team that are diverse and dynamic. In the proposed method, agents also benefit from predicting the roles of the agents in the opponent team. RAC uses the actor-critic framework with role encoder and opponent role predictors for learning an optimal policy. Experimentation using 2 games demonstrates that the policies learned by RAC achieve higher rewards than those learned using state-of-the-art baselines. Moreover, experiments suggest that the agents in a team learn diverse and opponent-aware policies.