Spatial Influence-aware Reinforcement Learning for Intelligent Transportation System

TL;DR

This paper introduces a spatial influence-aware multi-agent reinforcement learning approach for traffic light control in intelligent transportation systems, addressing scalability and coordination challenges in urban traffic management.

Contribution

It proposes a novel multi-agent DDPG system incorporating spatial influence and social utility, with empirical validation on grid road networks.

Findings

Spatial influence improves traffic flow efficiency.

Directed communication impacts network and individual utilities.

Selfish index affects overall group utility.

Abstract

Intelligent transportation systems (ITSs) are envisioned to be crucial for smart cities, which aims at improving traffic flow to improve the life quality of urban residents and reducing congestion to improve the efficiency of commuting. However, several challenges need to be resolved before such systems can be deployed, for example, conventional solutions for Markov decision process (MDP) and single-agent Reinforcement Learning (RL) algorithms suffer from poor scalability, and multi-agent systems suffer from poor communication and coordination. In this paper, we explore the potential of mutual information sharing, or in other words, spatial influence based communication, to optimize traffic light control policy. First, we mathematically analyze the transportation system. We conclude that the transportation system does not have stationary Nash Equilibrium, thereby reinforcement learning algorithms offer suitable solutions. Secondly, we describe how to build a multi-agent Deep Deterministic Policy Gradient (DDPG) system with spatial influence and social group utility incorporated. Then we utilize the grid topology road network to empirically demonstrate the scalability of the new system. We demonstrate three types of directed communications to show the effect of directions of social influence on the entire network utility and individual utility. Lastly, we define "selfish index" and analyze the effect of it on total group utility.