Multi-hop Upstream Anticipatory Traffic Signal Control with Deep Reinforcement Learning

TL;DR

This paper introduces a multi-hop upstream pressure concept for deep reinforcement learning-based traffic signal control, enabling broader spatial awareness and significantly reducing network delays in urban traffic management.

Contribution

It proposes a novel multi-hop upstream pressure metric based on Markov chain theory, enhancing the spatial awareness of traffic signal control agents in deep reinforcement learning.

Findings

Significantly reduces overall network delay in simulations.

Outperforms pressure-based methods focusing only on immediate upstream links.

Demonstrates effectiveness in both synthetic and real-world Toronto scenarios.

Abstract

Coordination in traffic signal control is crucial for managing congestion in urban networks. Existing pressure-based control methods focus only on immediate upstream links, leading to suboptimal green time allocation and increased network delays. However, effective signal control inherently requires coordination across a broader spatial scope, as the effect of upstream traffic should influence signal control decisions at downstream intersections, impacting a large area in the traffic network. Although agent communication using neural network-based feature extraction can implicitly enhance spatial awareness, it significantly increases the learning complexity, adding an additional layer of difficulty to the challenging task of control in deep reinforcement learning. To address the issue of learning complexity and myopic traffic pressure definition, our work introduces a novel concept based on Markov chain theory, namely \textit{multi-hop upstream pressure}, which generalizes the conventional pressure to account for traffic conditions beyond the immediate upstream links. This farsighted and compact metric informs the deep reinforcement learning agent to preemptively clear the multi-hop upstream queues, guiding the agent to optimize signal timings with a broader spatial awareness. Simulations on synthetic and realistic (Toronto) scenarios demonstrate controllers utilizing multi-hop upstream pressure significantly reduce overall network delay by prioritizing traffic movements based on a broader understanding of upstream congestion.