Distributionally Robust Multi-Agent Reinforcement Learning for Intelligent Traffic Control

TL;DR

This paper develops a distributionally robust multi-agent reinforcement learning framework for traffic signal control, improving performance under diverse and atypical traffic demand patterns by training on worst-case demand scenarios.

Contribution

It introduces a novel distributionally robust training method for multi-agent RL in traffic control, enhancing robustness against demand uncertainty without changing the controller architecture.

Findings

Up to 51% shorter queues in worst-case scenarios.

Up to 38% higher speeds in worst-case scenarios.

Consistent performance improvements across multiple demand scenarios.

Abstract

Learning-based traffic signal control is typically optimized for average performance under a few nominal demand patterns, which can result in poor behavior under atypical traffic conditions. To address this, we develop a distributionally robust multi-agent reinforcement learning framework for signal control on a 3x3 urban grid calibrated from a contiguous 3x3 subarea of central Athens covered by the pNEUMA trajectory dataset (Barmpounakis and Geroliminis, 2020). Our approach proceeds in three stages. First, we train a baseline multi-agent RL controller in which each intersection is governed by a proximal policy optimization agent with discrete signal phases, using a centralized training, decentralized execution paradigm. Second, to capture demand uncertainty, we construct eight heterogeneous origin-destination-based traffic scenarios-one directly derived from pNEUMA and seven synthetically generated-to span a wide range of spatial and temporal demand patterns. Over this scenario set, we train a contextual-bandit worst-case estimator that assigns mixture weights to estimate adversarial demand distributions conditioned on context. Finally, without modifying the controller architecture, we fine-tune the baseline multi-agent reinforcement learning agents under these estimated worst-case mixtures to obtain a distributionally robust multi-agent reinforcement learning controller. Across all eight scenarios, as well as on an unseen validation network based on the Sioux Falls configuration, the distributionally robust multi-agent reinforcement learning controller consistently reduces horizon-averaged queues and increases average speeds relative to the baseline, achieving up to 51% shorter queues and 38% higher speeds on the worst-performing scenarios.