Sequence Modeling for N-Agent Ad Hoc Teamwork

TL;DR

This paper introduces a transformer-based centralized method for N-agent ad hoc teamwork that improves collaboration with unseen teammates in partially observable environments, outperforming previous independent learning approaches.

Contribution

The work proposes a novel transformer-based approach for N-agent ad hoc teamwork, leveraging historical data to enhance collaboration and generalization.

Findings

MAT-NAHT outperforms POAM in StarCraft II tasks.

The approach achieves higher sample efficiency.

It generalizes better to unseen teammates.

Abstract

N-agent ad hoc teamwork (NAHT) is a newly introduced challenge in multi-agent reinforcement learning, where controlled subteams of varying sizes must dynamically collaborate with varying numbers and types of unknown teammates without pre-coordination. The existing learning algorithm (POAM) considers only independent learning for its flexibility in dealing with a changing number of agents. However, independent learning fails to fully capture the inter-agent dynamics essential for effective collaboration. Based on our observation that transformers deal effectively with sequences with varying lengths and have been shown to be highly effective for a variety of machine learning problems, this work introduces a centralized, transformer-based method for N-agent ad hoc teamwork. Our proposed approach incorporates historical observations and actions of all controlled agents, enabling optimal responses to diverse and unseen teammates in partially observable environments. Empirical evaluation on a StarCraft II task demonstrates that MAT-NAHT outperforms POAM, achieving superior sample efficiency and generalization, without auxiliary agent-modeling objectives.