Scalable Reinforcement Learning Policies for Multi-Agent Control

TL;DR

This paper introduces a scalable multi-agent reinforcement learning approach for target tracking that can handle thousands of pursuers and targets using decentralized control, attention mechanisms, and heuristic training methods.

Contribution

It presents a novel scalable MARL framework with attention-based value functions and heuristics for training on smaller problems, enabling large-scale multi-agent control.

Findings

Successfully tracked up to 1000 pursuers and targets in simulations.

Demonstrated robustness and scalability compared to existing algorithms.

Showed effective decentralized control with weak cooperation among agents.

Abstract

We develop a Multi-Agent Reinforcement Learning (MARL) method to learn scalable control policies for target tracking. Our method can handle an arbitrary number of pursuers and targets; we show results for tasks consisting up to 1000 pursuers tracking 1000 targets. We use a decentralized, partially-observable Markov Decision Process framework to model pursuers as agents receiving partial observations (range and bearing) about targets which move using fixed, unknown policies. An attention mechanism is used to parameterize the value function of the agents; this mechanism allows us to handle an arbitrary number of targets. Entropy-regularized off-policy RL methods are used to train a stochastic policy, and we discuss how it enables a hedging behavior between pursuers that leads to a weak form of cooperation in spite of completely decentralized control execution. We further develop a masking heuristic that allows training on smaller problems with few pursuers-targets and execution on much larger problems. Thorough simulation experiments, ablation studies, and comparisons to state of the art algorithms are performed to study the scalability of the approach and robustness of performance to varying numbers of agents and targets.