Submodular Multi-Agent Policy Learning for Online Distributed Task Allocation in Open Multi-Agent Systems

TL;DR

This paper introduces a novel continuous relaxation called Partition Multilinear Extension for multi-agent reinforcement learning, enabling effective online distributed task allocation with theoretical guarantees and practical GNN-based implementations.

Contribution

It proposes the PME relaxation, a submodular difference reward-based policy-gradient framework, with proven approximation guarantees and applicability to open multi-agent systems.

Findings

SubMAPG outperforms local greedy and shared-reward baselines in experiments.

Theoretical guarantees include a 1/2-approximation and sublinear regret.

GNN policies effectively handle open systems with dynamic agents and targets.

Abstract

This paper studies multi-agent reinforcement learning with submodular team utilities for online distributed task allocation. In this setting, each agent selects one action from a local categorical policy, so feasible joint actions form a partition matroid over agent-action pairs. Classical multilinear extensions use independent Bernoulli sampling and therefore do not match the categorical policies executed by decentralized agents. To address this mismatch, we introduce the Partition Multilinear Extension (PME), a continuous relaxation whose value equals the expected team utility under factorized categorical policies. We prove that submodular difference rewards provide unbiased PME marginal-gradient information and yield a stagewise score-function policy-gradient estimator. Based on this connection, we propose SubMAPG, a centralized-training decentralized-execution policy-gradient framework with masked categorical policies and submodular difference-reward training signals. For the associated PME marginal-space projected stochastic-gradient dynamics, we prove a stagewise 1/2-approximation guarantee and sublinear dynamic regret in slowly varying environments, measured by the path length of the optimal PME marginals. To handle open systems with time-varying agents and targets, we instantiate SubMAPG with graph neural network policies. Experiments on multi-robot coverage and multi-target tracking show that SubMAPG outperforms local greedy and shared-reward baselines and is competitive with centralized myopic greedy strategies.