Moving Forward in Formation: A Decentralized Hierarchical Learning Approach to Multi-Agent Moving Together

TL;DR

This paper introduces a decentralized hierarchical reinforcement learning approach for multi-agent formation pathfinding, enabling scalable, cooperative navigation in complex environments with real-world validation.

Contribution

It proposes a novel hierarchical RL algorithm with a theoretical weighting scheme and communication strategy for multi-agent formation tasks, addressing limitations of prior centralized and RL methods.

Findings

Outperforms existing RL methods in simulation

Scales effectively to large environments

Successfully validated in real-world scenarios

Abstract

Multi-agent path finding in formation has many potential real-world applications like mobile warehouse robots. However, previous multi-agent path finding (MAPF) methods hardly take formation into consideration. Furthermore, they are usually centralized planners and require the whole state of the environment. Other decentralized partially observable approaches to MAPF are reinforcement learning (RL) methods. However, these RL methods encounter difficulties when learning path finding and formation problem at the same time. In this paper, we propose a novel decentralized partially observable RL algorithm that uses a hierarchical structure to decompose the multi objective task into unrelated ones. It also calculates a theoretical weight that makes every task reward has equal influence on the final RL value function. Additionally, we introduce a communication method that helps agents cooperate with each other. Experiments in simulation show that our method outperforms other end-to-end RL methods and our method can naturally scale to large world sizes where centralized planner struggles. We also deploy and validate our method in a real world scenario.