Feudal Multi-Agent Reinforcement Learning with Adaptive Network Partition for Traffic Signal Control

TL;DR

This paper introduces an adaptive feudal multi-agent reinforcement learning approach for traffic signal control, dynamically partitioning traffic networks to improve cooperation and traffic flow in real-time scenarios.

Contribution

It proposes a novel adaptive network partition method using GNN and MCTS, enhancing multi-agent cooperation in traffic signal control over static partitioning methods.

Findings

Achieves better average travel time than existing methods.

Reduces queue length effectively in real-world traffic networks.

Demonstrates adaptability to dynamic traffic flow.

Abstract

Multi-agent reinforcement learning (MARL) has been applied and shown great potential in multi-intersections traffic signal control, where multiple agents, one for each intersection, must cooperate together to optimize traffic flow. To encourage global cooperation, previous work partitions the traffic network into several regions and learns policies for agents in a feudal structure. However, static network partition fails to adapt to dynamic traffic flow, which will changes frequently over time. To address this, we propose a novel feudal MARL approach with adaptive network partition. Specifically, we first partition the network into several regions according to the traffic flow. To do this, we propose two approaches: one is directly to use graph neural network (GNN) to generate the network partition, and the other is to use Monte-Carlo tree search (MCTS) to find the best partition with criteria computed by GNN. Then, we design a variant of Qmix using GNN to handle various dimensions of input, given by the dynamic network partition. Finally, we use a feudal hierarchy to manage agents in each partition and promote global cooperation. By doing so, agents are able to adapt to the traffic flow as required in practice. We empirically evaluate our method both in a synthetic traffic grid and real-world traffic networks of three cities, widely used in the literature. Our experimental results confirm that our method can achieve better performance, in terms of average travel time and queue length, than several leading methods for traffic signal control.