iLLM-TSC: Integration reinforcement learning and large language model for traffic signal control policy improvement

TL;DR

This paper introduces a novel framework combining large language models with reinforcement learning to improve traffic signal control, especially under communication issues and rare events, significantly reducing average waiting times.

Contribution

The paper presents an innovative integration of LLMs with RL for traffic signal control, addressing communication imperfections and rare events without modifying existing RL systems.

Findings

Reduces average waiting time by 17.5% under degraded communication conditions

Enhances RL policy robustness by verifying decisions with LLMs

Seamlessly integrates with existing RL-based TSC systems

Abstract

Urban congestion remains a critical challenge, with traffic signal control (TSC) emerging as a potent solution. TSC is often modeled as a Markov Decision Process problem and then solved using reinforcement learning (RL), which has proven effective. However, the existing RL-based TSC system often overlooks imperfect observations caused by degraded communication, such as packet loss, delays, and noise, as well as rare real-life events not included in the reward function, such as unconsidered emergency vehicles. To address these limitations, we introduce a novel integration framework that combines a large language model (LLM) with RL. This framework is designed to manage overlooked elements in the reward function and gaps in state information, thereby enhancing the policies of RL agents. In our approach, RL initially makes decisions based on observed data. Subsequently, LLMs evaluate these decisions to verify their reasonableness. If a decision is found to be unreasonable, it is adjusted accordingly. Additionally, this integration approach can be seamlessly integrated with existing RL-based TSC systems without necessitating modifications. Extensive testing confirms that our approach reduces the average waiting time by $17.5\%$ in degraded communication conditions as compared to traditional RL methods, underscoring its potential to advance practical RL applications in intelligent transportation systems. The related code can be found at \url{https://github.com/Traffic-Alpha/iLLM-TSC}.