Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learning

TL;DR

This paper introduces a prompt-based method leveraging Large Language Models to understand real-world traffic dynamics, improving the transfer of reinforcement learning policies from simulation to actual traffic environments.

Contribution

It proposes a novel prompt learning approach using LLMs to profile system dynamics, enhancing sim-to-real transfer in traffic signal control with reinforcement learning.

Findings

LLM-based prompt method reduces sim-to-real performance gap

Experimental results show improved policy effectiveness in real-world scenarios

Method outperforms traditional transfer techniques in traffic signal control

Abstract

Numerous solutions are proposed for the Traffic Signal Control (TSC) tasks aiming to provide efficient transportation and mitigate congestion waste. In recent, promising results have been attained by Reinforcement Learning (RL) methods through trial and error in simulators, bringing confidence in solving cities' congestion headaches. However, there still exist performance gaps when simulator-trained policies are deployed to the real world. This issue is mainly introduced by the system dynamic difference between the training simulator and the real-world environments. The Large Language Models (LLMs) are trained on mass knowledge and proved to be equipped with astonishing inference abilities. In this work, we leverage LLMs to understand and profile the system dynamics by a prompt-based grounded action transformation. Accepting the cloze prompt template, and then filling in the answer based on accessible context, the pre-trained LLM's inference ability is exploited and applied to understand how weather conditions, traffic states, and road types influence traffic dynamics, being aware of this, the policies' action is taken and grounded based on realistic dynamics, thus help the agent learn a more realistic policy. We conduct experiments using DQN to show the effectiveness of the proposed PromptGAT's ability in mitigating the performance gap from simulation to reality (sim-to-real).