Cooperative Reinforcement Learning on Traffic Signal Control

TL;DR

This paper introduces COMMA-DDPG, a cooperative multi-agent reinforcement learning framework for traffic signal control that significantly reduces total delay time by coordinating local and global agents.

Contribution

It proposes a novel multi-objective, cooperative RL architecture with age-decaying weights, enabling better traffic signal optimization and faster learning.

Findings

Reduces total delay time by 60% compared to state-of-the-art methods.

Effective coordination between local and global agents improves traffic flow.

Proven convergence and solution existence for the proposed RL approach.

Abstract

Traffic signal control is a challenging real-world problem aiming to minimize overall travel time by coordinating vehicle movements at road intersections. Existing traffic signal control systems in use still rely heavily on oversimplified information and rule-based methods. Specifically, the periodicity of green/red light alternations can be considered as a prior for better planning of each agent in policy optimization. To better learn such adaptive and predictive priors, traditional RL-based methods can only return a fixed length from predefined action pool with only local agents. If there is no cooperation between these agents, some agents often make conflicts to other agents and thus decrease the whole throughput. This paper proposes a cooperative, multi-objective architecture with age-decaying weights to better estimate multiple reward terms for traffic signal control optimization, which termed COoperative Multi-Objective Multi-Agent Deep Deterministic Policy Gradient (COMMA-DDPG). Two types of agents running to maximize rewards of different goals - one for local traffic optimization at each intersection and the other for global traffic waiting time optimization. The global agent is used to guide the local agents as a means for aiding faster learning but not used in the inference phase. We also provide an analysis of solution existence together with convergence proof for the proposed RL optimization. Evaluation is performed using real-world traffic data collected using traffic cameras from an Asian country. Our method can effectively reduce the total delayed time by 60\%. Results demonstrate its superiority when compared to SoTA methods.