Theory of Mind Guided Strategy Adaptation for Zero-Shot Coordination

TL;DR

This paper introduces a Theory-of-Mind guided adaptive ensemble approach for zero-shot coordination in multi-agent reinforcement learning, enabling agents to better infer teammates' intentions and adapt their strategies accordingly.

Contribution

It proposes a novel adaptive ensemble agent that leverages Theory-of-Mind for best-response selection, improving zero-shot coordination performance over static policies.

Findings

Outperforms baseline in Overcooked environment

Effective in both fully and partially observable settings

Demonstrates improved adaptability and synergy

Abstract

A central challenge in multi-agent reinforcement learning is enabling agents to adapt to previously unseen teammates in a zero-shot fashion. Prior work in zero-shot coordination often follows a two-stage process, first generating a diverse training pool of partner agents, and then training a best-response agent to collaborate effectively with the entire training pool. While many previous works have achieved strong performance by devising better ways to diversify the partner agent pool, there has been less emphasis on how to leverage this pool to build an adaptive agent. One limitation is that the best-response agent may converge to a static, generalist policy that performs reasonably well across diverse teammates, rather than learning a more adaptive, specialist policy that can better adapt to teammates and achieve higher synergy. To address this, we propose an adaptive ensemble agent that uses Theory-of-Mind-based best-response selection to first infer its teammate's intentions and then select the most suitable policy from a policy ensemble. We conduct experiments in the Overcooked environment to evaluate zero-shot coordination performance under both fully and partially observable settings. The empirical results demonstrate the superiority of our method over a single best-response baseline.