Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Games

TL;DR

This paper introduces noisy zero-shot coordination (NZSC), a framework where agents learn to coordinate despite observing noisy versions of the environment, addressing the limitations of the common knowledge assumption in real-world settings.

Contribution

The paper formulates NZSC, reduces it to ZSC via a meta-Dec-POMDP, and proposes a meta-learning training method enabling RL agents to coordinate with novel partners under noisy observations.

Findings

NZSC training enables effective coordination with noisy environment observations.

Agents trained with NZSC perform well with unseen partners in noisy settings.

The approach relaxes the common knowledge assumption, making coordination more robust in real-world scenarios.

Abstract

Zero-shot coordination (ZSC) is a popular setting for studying the ability of reinforcement learning (RL) agents to coordinate with novel partners. Prior ZSC formulations assume the $\textit{problem setting}$ is common knowledge: each agent knows the underlying Dec-POMDP, knows others have this knowledge, and so on ad infinitum. However, this assumption rarely holds in complex real-world settings, which are often difficult to fully and correctly specify. Hence, in settings where this common knowledge assumption is invalid, agents trained using ZSC methods may not be able to coordinate well. To address this limitation, we formulate the $\textit{noisy zero-shot coordination}$ (NZSC) problem. In NZSC, agents observe different noisy versions of the ground truth Dec-POMDP, which are assumed to be distributed according to a fixed noise model. Only the distribution of ground truth Dec-POMDPs and the noise model are common knowledge. We show that a NZSC problem can be reduced to a ZSC problem by designing a meta-Dec-POMDP with an augmented state space consisting of all the ground-truth Dec-POMDPs. For solving NZSC problems, we propose a simple and flexible meta-learning method called NZSC training, in which the agents are trained across a distribution of coordination problems - which they only get to observe noisy versions of. We show that with NZSC training, RL agents can be trained to coordinate well with novel partners even when the (exact) problem setting of the coordination is not common knowledge.