Analyzing Robustness of the Deep Reinforcement Learning Algorithm in Ramp Metering Applications Considering False Data Injection Attack and Defense

TL;DR

This paper develops a deep Q-learning-based ramp metering control method that is robust against false data injection and noise attacks, outperforming traditional ALINEA control in diverse real-world scenarios.

Contribution

It introduces a robust deep reinforcement learning model for ramp metering using only loop detector data, tested under attack scenarios across different environments.

Findings

The proposed model outperforms ALINEA in robustness and efficiency.

It maintains high performance under false data injection and noise attacks.

The model is adaptable to various road geometries and demands.

Abstract

Ramp metering is the act of controlling on-going vehicles to the highway mainlines. Decades of practices of ramp metering have proved that ramp metering can decrease total travel time, mitigate shockwaves, decrease rear-end collisions by smoothing the traffic interweaving process, etc. Besides traditional control algorithm like ALINEA, Deep Reinforcement Learning (DRL) algorithms have been introduced to build a finer control. However, two remaining challenges still hinder DRL from being implemented in the real world: (1) some assumptions of algorithms are hard to be matched in the real world; (2) the rich input states may make the model vulnerable to attacks and data noises. To investigate these issues, we propose a Deep Q-Learning algorithm using only loop detectors information as inputs in this study. Then, a set of False Data Injection attacks and random noise attack are designed to investigate the robustness of the model. The major benefit of the model is that it can be applied to almost any ramp metering sites regardless of the road geometries and layouts. Besides outcompeting the ALINEA method, the Deep Q-Learning method also shows a good robustness through training among very different demands and geometries. For example, during the testing case in I-24 near Murfreesboro, TN, the model shows its robustness as it still outperforms ALINEA algorithm under Fast Gradient Sign Method attacks. Unlike many previous studies, the model is trained and tested in completely different environments to show the capabilities of the model.