Toward Dependency Dynamics in Multi-Agent Reinforcement Learning for Traffic Signal Control

TL;DR

This paper explores how multi-agent reinforcement learning can adaptively optimize traffic signals by considering dependency dynamics among intersections, proposing a novel update strategy that improves convergence without losing optimality.

Contribution

It introduces a dependency-aware dynamic parameter update strategy for deep Q-networks in multi-agent traffic control, bridging centralized and decentralized RL approaches based on congestion conditions.

Findings

DQN-DPUS accelerates convergence in traffic signal control tasks.

The strategy maintains optimal exploration while improving learning speed.

Theoretical analysis confirms the effectiveness of dependency-based updates.

Abstract

Reinforcement learning (RL) emerges as a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, with deep neural networks substantially augmenting its learning capabilities. However, centralized RL becomes impractical for ATSC involving multiple agents due to the exceedingly high dimensionality of the joint action space. Multi-agent RL (MARL) mitigates this scalability issue by decentralizing control to local RL agents. Nevertheless, this decentralized method introduces new challenges: the environment becomes partially observable from the perspective of each local agent due to constrained inter-agent communication. Both centralized RL and MARL exhibit distinct strengths and weaknesses, particularly under heavy intersectional traffic conditions. In this paper, we justify that MARL can achieve the optimal global Q-value by separating into multiple IRL (Independent Reinforcement Learning) processes when no spill-back congestion occurs (no agent dependency) among agents (intersections). In the presence of spill-back congestion (with agent dependency), the maximum global Q-value can be achieved by using centralized RL. Building upon the conclusions, we propose a novel Dynamic Parameter Update Strategy for Deep Q-Network (DQN-DPUS), which updates the weights and bias based on the dependency dynamics among agents, i.e. updating only the diagonal sub-matrices for the scenario without spill-back congestion. We validate the DQN-DPUS in a simple network with two intersections under varying traffic, and show that the proposed strategy can speed up the convergence rate without sacrificing optimal exploration. The results corroborate our theoretical findings, demonstrating the efficacy of DQN-DPUS in optimizing traffic signal control.