Learning Decentralized Traffic Signal Controllers with Multi-Agent Graph Reinforcement Learning

TL;DR

This paper introduces a novel decentralized traffic signal control method using multi-agent graph reinforcement learning, leveraging topology-aware information aggregation and diffusion convolution to improve adaptivity and scalability in smart city traffic management.

Contribution

It proposes a new MARL architecture with a topology-aware information aggregation strategy and diffusion convolution, enhancing spatial-temporal correlation capture for decentralized traffic control.

Findings

Outperforms existing decentralized algorithms in experiments

Effective in both synthetic and real-world datasets

Improves environmental observability and learning capacity

Abstract

This paper considers optimal traffic signal control in smart cities, which has been taken as a complex networked system control problem. Given the interacting dynamics among traffic lights and road networks, attaining controller adaptivity and scalability stands out as a primary challenge. Capturing the spatial-temporal correlation among traffic lights under the framework of Multi-Agent Reinforcement Learning (MARL) is a promising solution. Nevertheless, existing MARL algorithms ignore effective information aggregation which is fundamental for improving the learning capacity of decentralized agents. In this paper, we design a new decentralized control architecture with improved environmental observability to capture the spatial-temporal correlation. Specifically, we first develop a topology-aware information aggregation strategy to extract correlation-related information from unstructured data gathered in the road network. Particularly, we transfer the road network topology into a graph shift operator by forming a diffusion process on the topology, which subsequently facilitates the construction of graph signals. A diffusion convolution module is developed, forming a new MARL algorithm, which endows agents with the capabilities of graph learning. Extensive experiments based on both synthetic and real-world datasets verify that our proposal outperforms existing decentralized algorithms.