Bridging the Reality Gap of Reinforcement Learning based Traffic Signal Control using Domain Randomization and Meta Learning

TL;DR

This paper investigates how to reduce the discrepancy between simulation and real-world traffic signal control using domain randomization and meta-learning, demonstrating improved transferability of RL policies across different simulation models.

Contribution

It provides a comprehensive analysis of simulation parameters affecting the reality gap and evaluates two strategies, DR and MAML, for bridging this gap in RL-based traffic signal control.

Findings

Both DR and MAML outperform existing RL algorithms in cross-simulator tests.

The study highlights the effectiveness of domain randomization and meta-learning in real-world deployment scenarios.

Experimental results confirm the potential of these methods to mitigate the reality gap.

Abstract

Reinforcement Learning (RL) has been widely explored in Traffic Signal Control (TSC) applications, however, still no such system has been deployed in practice. A key barrier to progress in this area is the reality gap, the discrepancy that results from differences between simulation models and their real-world equivalents. In this paper, we address this challenge by first presenting a comprehensive analysis of potential simulation parameters that contribute to this reality gap. We then also examine two promising strategies that can bridge this gap: Domain Randomization (DR) and Model-Agnostic Meta-Learning (MAML). Both strategies were trained with a traffic simulation model of an intersection. In addition, the model was embedded in LemgoRL, a framework that integrates realistic, safety-critical requirements into the control system. Subsequently, we evaluated the performance of the two methods on a separate model of the same intersection that was developed with a different traffic simulator. In this way, we mimic the reality gap. Our experimental results show that both DR and MAML outperform a state-of-the-art RL algorithm, therefore highlighting their potential to mitigate the reality gap in RLbased TSC systems.