Cooperative Path Planning with Asynchronous Multiagent Reinforcement Learning

TL;DR

This paper introduces an asynchronous multi-agent reinforcement learning framework for cooperative path planning in multi-source-destination shortest path problems, improving efficiency and cooperation among vehicles in complex road networks.

Contribution

It proposes a novel asyn-MARL framework with a global state, trajectory collection, and specialized actor network to enhance multi-agent cooperation and efficiency in route planning.

Findings

Outperforms existing planning methods on synthetic and real networks.

Reduces training trajectory redundancy and improves cooperation.

Effectively prevents infinite loops in routing paths.

Abstract

In this paper, we study the shortest path problem (SPP) with multiple source-destination pairs (MSD), namely MSD-SPP, to minimize average travel time of all shortest paths. The inherent traffic capacity limits within a road network contributes to the competition among vehicles. Multi-agent reinforcement learning (MARL) model cannot offer effective and efficient path planning cooperation due to the asynchronous decision making setting in MSD-SPP, where vehicles (a.k.a agents) cannot simultaneously complete routing actions in the previous time step. To tackle the efficiency issue, we propose to divide an entire road network into multiple sub-graphs and subsequently execute a two-stage process of inter-region and intra-region route planning. To address the asynchronous issue, in the proposed asyn-MARL framework, we first design a global state, which exploits a low-dimensional vector to implicitly represent the joint observations and actions of multi-agents. Then we develop a novel trajectory collection mechanism to decrease the redundancy in training trajectories. Additionally, we design a novel actor network to facilitate the cooperation among vehicles towards the same or close destinations and a reachability graph aimed at preventing infinite loops in routing paths. On both synthetic and real road networks, our evaluation result demonstrates that our approach outperforms state-of-the-art planning approaches.