Uncertainty-aware Grounded Action Transformation towards Sim-to-Real Transfer for Traffic Signal Control

TL;DR

This paper introduces UGAT, a novel method for transferring reinforcement learning policies for traffic signal control from simulation to real-world environments by dynamically transforming actions with uncertainty to reduce domain gap.

Contribution

The paper presents UGAT, a new sim-to-real transfer approach that incorporates uncertainty-aware action transformation to improve RL policy transfer in traffic signal control.

Findings

Significant performance improvement in real-world traffic environments

Effective reduction of simulation-to-real domain gap

Demonstrated success in simulated traffic scenarios

Abstract

Traffic signal control (TSC) is a complex and important task that affects the daily lives of millions of people. Reinforcement Learning (RL) has shown promising results in optimizing traffic signal control, but current RL-based TSC methods are mainly trained in simulation and suffer from the performance gap between simulation and the real world. In this paper, we propose a simulation-to-real-world (sim-to-real) transfer approach called UGAT, which transfers a learned policy trained from a simulated environment to a real-world environment by dynamically transforming actions in the simulation with uncertainty to mitigate the domain gap of transition dynamics. We evaluate our method on a simulated traffic environment and show that it significantly improves the performance of the transferred RL policy in the real world.