Batch-Augmented Multi-Agent Reinforcement Learning for Efficient Traffic Signal Optimization

TL;DR

This paper introduces a novel two-stage reinforcement learning framework for traffic signal control that is robust to data limitations and reduces congestion efficiently without extensive simulation.

Contribution

It proposes a batch-augmented multi-agent reinforcement learning approach with a two-stage framework, including an Evolution Strategies method and off-policy learning components, addressing real-world traffic control challenges.

Findings

Reduced traffic congestion by 36% in waiting time

Achieved effective control with only 600 simulator queries

Handled data loss and learning without shared intersection information

Abstract

The goal of this work is to provide a viable solution based on reinforcement learning for traffic signal control problems. Although the state-of-the-art reinforcement learning approaches have yielded great success in a variety of domains, directly applying it to alleviate traffic congestion can be challenging, considering the requirement of high sample efficiency and how training data is gathered. In this work, we address several challenges that we encountered when we attempted to mitigate serious traffic congestion occurring in a metropolitan area. Specifically, we are required to provide a solution that is able to (1) handle the traffic signal control when certain surveillance cameras that retrieve information for reinforcement learning are down, (2) learn from batch data without a traffic simulator, and (3) make control decisions without shared information across intersections. We present a two-stage framework to deal with the above-mentioned situations. The framework can be decomposed into an Evolution Strategies approach that gives a fixed-time traffic signal control schedule and a multi-agent off-policy reinforcement learning that is capable of learning from batch data with the aid of three proposed components, bounded action, batch augmentation, and surrogate reward clipping. Our experiments show that the proposed framework reduces traffic congestion by 36% in terms of waiting time compared with the currently used fixed-time traffic signal plan. Furthermore, the framework requires only 600 queries to a simulator to achieve the result.