Area-wide traffic signal control based on a deep graph Q-Network (DGQN) trained in an asynchronous manner

TL;DR

This paper introduces a deep graph Q-network (DGQN) trained asynchronously to optimize traffic signal control across large transportation networks, effectively capturing spatio-temporal dependencies and improving convergence speed.

Contribution

The paper presents a novel DGQN architecture with an asynchronous training method for large-scale traffic signal control, addressing action space complexity and spatio-temporal correlation challenges.

Findings

DGQN outperforms existing RL algorithms in traffic control tasks.

Asynchronous training accelerates convergence to optimal policies.

The approach improves traffic flow in Seoul's transportation network.

Abstract

Reinforcement learning (RL) algorithms have been widely applied in traffic signal studies. There are, however, several problems in jointly controlling traffic lights for a large transportation network. First, the action space exponentially explodes as the number of intersections to be jointly controlled increases. Although a multi-agent RL algorithm has been used to solve the curse of dimensionality, this neither guaranteed a global optimum, nor could it break the ties between joint actions. The problem was circumvented by revising the output structure of a deep Q-network (DQN) within the framework of a single-agent RL algorithm. Second, when mapping traffic states into an action value, it is difficult to consider spatio-temporal correlations over a large transportation network. A deep graph Q-network (DGQN) was devised to efficiently accommodate spatio-temporal dependencies on a large scale. Finally, training a RL model to jointly control traffic lights in a large transportation network requires much time to converge. An asynchronous update methodology was devised for a DGQN to quickly reach an optimal policy. Using these three remedies, a DGQN succeeded in jointly controlling the traffic lights in a large transportation network in Seoul. This approach outperformed other state-of-the-art RL algorithms as well as an actual fixed-signal operation.