Principled Learning-to-Communicate with Quasi-Classical Information Structures

TL;DR

This paper formalizes and analyzes learning-to-communicate in multi-agent systems using information structures, focusing on quasi-classical cases to develop efficient algorithms and understand computational complexity.

Contribution

It introduces a formal framework for LTC with information structures, identifies conditions for tractability, and proposes algorithms with provable complexity bounds.

Findings

Non-classical LTCs are generally computationally intractable.

Proposed conditions ensure LTC preserves quasi-classical information structures.

Developed algorithms with quasi-polynomial time and sample complexities for certain LTC problems.

Abstract

Learning-to-communicate (LTC) in partially observable environments has received increasing attention in deep multi-agent reinforcement learning, where the control and communication strategies are jointly learned. Meanwhile, the impact of communication on decision-making has been extensively studied in control theory. In this paper, we seek to formalize and better understand LTC by bridging these two lines of work, through the lens of information structures (ISs). To this end, we formalize LTC in decentralized partially observable Markov decision processes (Dec-POMDPs) under the common-information-based framework from decentralized stochastic control, and classify LTC problems based on the ISs before (additional) information sharing. We first show that non-classical LTCs are computationally intractable in general, and thus focus on quasi-classical (QC) LTCs. We then propose a series of conditions for QC LTCs, under which LTC preserves the QC IS after information sharing, whereas violating them can cause computational hardness in general. Further, we develop provable planning and learning algorithms for QC LTCs, and establish quasi-polynomial time and sample complexities for several QC LTC examples that satisfy the above conditions. Along the way, we also establish new results on a relationship between (strictly) QC IS and the condition of having strategy-independent common-information-based beliefs (SI-CIBs), as well as on solving Dec-POMDPs without computationally intractable oracles but beyond those with SI-CIBs, which may be of independent interest.