Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning

TL;DR

HOP is a hierarchical multi-agent reinforcement learning algorithm that enables rapid adaptation to unseen opponents by combining opponent modeling with Monte Carlo Tree Search planning.

Contribution

The paper introduces HOP, a novel hierarchical approach that improves few-shot adaptation in mixed-motive environments through integrated opponent modeling and planning.

Findings

HOP outperforms existing methods in few-shot adaptation scenarios.

HOP demonstrates superior performance in self-play environments.

Emergence of social intelligence observed during experiments.

Abstract

Despite the recent successes of multi-agent reinforcement learning (MARL) algorithms, efficiently adapting to co-players in mixed-motive environments remains a significant challenge. One feasible approach is to hierarchically model co-players' behavior based on inferring their characteristics. However, these methods often encounter difficulties in efficient reasoning and utilization of inferred information. To address these issues, we propose Hierarchical Opponent modeling and Planning (HOP), a novel multi-agent decision-making algorithm that enables few-shot adaptation to unseen policies in mixed-motive environments. HOP is hierarchically composed of two modules: an opponent modeling module that infers others' goals and learns corresponding goal-conditioned policies, and a planning module that employs Monte Carlo Tree Search (MCTS) to identify the best response. Our approach improves efficiency by updating beliefs about others' goals both across and within episodes and by using information from the opponent modeling module to guide planning. Experimental results demonstrate that in mixed-motive environments, HOP exhibits superior few-shot adaptation capabilities when interacting with various unseen agents, and excels in self-play scenarios. Furthermore, the emergence of social intelligence during our experiments underscores the potential of our approach in complex multi-agent environments.