Toward multi-target self-organizing pursuit in a partially observable Markov game

TL;DR

This paper introduces FSC2, a decentralized multi-agent framework using deep reinforcement learning and fuzzy task allocation to improve multi-target pursuit in partially observable environments, achieving high efficiency and scalability.

Contribution

It proposes a novel distributed algorithm combining MARL and fuzzy task allocation for multi-target pursuit in POMG, addressing coordination, scalability, and partial observation challenges.

Findings

FSC2 outperforms other implicit coordination policies in experiments.

Scalability demonstrated with up to 2048 agents achieving near 100% success.

Component algorithms are validated for interpretability and effectiveness.

Abstract

The multiple-target self-organizing pursuit (SOP) problem has wide applications and has been considered a challenging self-organization game for distributed systems, in which intelligent agents cooperatively pursue multiple dynamic targets with partial observations. This work proposes a framework for decentralized multi-agent systems to improve the implicit coordination capabilities in search and pursuit. We model a self-organizing system as a partially observable Markov game (POMG) featured by large-scale, decentralization, partial observation, and noncommunication. The proposed distributed algorithm: fuzzy self-organizing cooperative coevolution (FSC2) is then leveraged to resolve the three challenges in multi-target SOP: distributed self-organizing search (SOS), distributed task allocation, and distributed single-target pursuit. FSC2 includes a coordinated multi-agent deep reinforcement learning (MARL) method that enables homogeneous agents to learn natural SOS patterns. Additionally, we propose a fuzzy-based distributed task allocation method, which locally decomposes multi-target SOP into several single-target pursuit problems. The cooperative coevolution principle is employed to coordinate distributed pursuers for each single-target pursuit problem. Therefore, the uncertainties of inherent partial observation and distributed decision-making in the POMG can be alleviated. The experimental results demonstrate that by decomposing the SOP task, FSC2 achieves superior performance compared with other implicit coordination policies fully trained by general MARL algorithms. The scalability of FSC2 is proved that up to 2048 FSC2 agents perform efficient multi-target SOP with almost 100 percent capture rates. Empirical analyses and ablation studies verify the interpretability, rationality, and effectiveness of component algorithms in FSC2.