Expected Return Symmetries

TL;DR

This paper introduces expected return symmetries, a broader class of symmetries in multi-agent environments, enabling agents to achieve better coordination without prior environment knowledge.

Contribution

It proposes expected return symmetries, expanding the types of symmetries used for coordination, and demonstrates improved zero-shot coordination results.

Findings

Agents trained with expected return symmetries outperform those with environment symmetries.

The method requires minimal prior assumptions about the environment.

Expected return symmetries encompass environment symmetries as a subgroup.

Abstract

Symmetry is an important inductive bias that can improve model robustness and generalization across many deep learning domains. In multi-agent settings, a priori known symmetries have been shown to address a fundamental coordination failure mode known as mutually incompatible symmetry breaking; e.g. in a game where two independent agents can choose to move "left'' or "right'', and where a reward of +1 or -1 is received when the agents choose the same action or different actions, respectively. However, the efficient and automatic discovery of environment symmetries, in particular for decentralized partially observable Markov decision processes, remains an open problem. Furthermore, environmental symmetry breaking constitutes only one type of coordination failure, which motivates the search for a more accessible and broader symmetry class. In this paper, we introduce such a broader group of previously unexplored symmetries, which we call expected return symmetries, which contains environment symmetries as a subgroup. We show that agents trained to be compatible under the group of expected return symmetries achieve better zero-shot coordination results than those using environment symmetries. As an additional benefit, our method makes minimal a priori assumptions about the structure of their environment and does not require access to ground truth symmetries.