Back to Basics: Deep Reinforcement Learning in Traffic Signal Control

TL;DR

This paper introduces RLight, a robust deep reinforcement learning approach for traffic signal control that improves learning speed, generalizes well to unseen traffic, and outperforms existing methods on real-world data.

Contribution

The paper presents a novel lightweight, cluster-aware state representation, a reformulated MDP that speeds up learning, and insights into phase transition actions, advancing RL-based traffic control.

Findings

RLight outperforms state-of-the-art algorithms on Hangzhou traffic data.

Speeded up learning by 30% through MDP reformulation.

Demonstrated good generalization to unseen traffic flows.

Abstract

In this paper we revisit some of the fundamental premises for a reinforcement learning (RL) approach to self-learning traffic lights. We propose RLight, a combination of choices that offers robust performance and good generalization to unseen traffic flows. In particular, our main contributions are threefold: our lightweight and cluster-aware state representation leads to improved performance; we reformulate the Markov Decision Process (MDP) such that it skips redundant timesteps of yellow light, speeding up learning by 30%; and we investigate the action space and provide insight into the difference in performance between acyclic and cyclic phase transitions. Additionally, we provide insights into the generalisation of the methods to unseen traffic. Evaluations using the real-world Hangzhou traffic dataset show that RLight outperforms state-of-the-art rule-based and deep reinforcement learning algorithms, demonstrating the potential of RL-based methods to improve urban traffic flows.