Cooperative Multi-Agent Assignment over Stochastic Graphs via Constrained Reinforcement Learning

TL;DR

This paper introduces a novel multi-agent reinforcement learning approach for dynamic task coordination over stochastic networks, allowing agents to adapt policies in real-time without requiring dual variable convergence.

Contribution

It proposes a new formulation where dual variables cycle, enabling scalable, feasible multi-agent coordination with limited communication and bounded estimation errors.

Findings

Agents achieve almost sure feasibility in dynamic environments

The method works with stochastic, time-varying network connectivity

Numerical experiments demonstrate successful multi-robot patrols

Abstract

Constrained multi-agent reinforcement learning offers the framework to design scalable and almost surely feasible solutions for teams of agents operating in dynamic environments to carry out conflicting tasks. We address the challenges of multi-agent coordination through an unconventional formulation in which the dual variables are not driven to convergence but are free to cycle, enabling agents to adapt their policies dynamically based on real-time constraint satisfaction levels. The coordination relies on a light single-bit communication protocol over a network with stochastic connectivity. Using this gossiped information, agents update local estimates of the dual variables. Furthermore, we modify the local dual dynamics by introducing a contraction factor, which lets us use finite communication buffers and keep the estimation error bounded. Under this model, we provide theoretical guarantees of almost sure feasibility and corroborate them with numerical experiments in which a team of robots successfully patrols multiple regions, communicating under a time-varying ad-hoc network.