Data efficient reinforcement learning and adaptive optimal perimeter control of network traffic dynamics

TL;DR

This paper introduces an integral reinforcement learning approach for adaptive, data-efficient, and robust perimeter traffic control that learns system dynamics without requiring precise models, improving real-time traffic management.

Contribution

The work develops a model-free IRL-based control method with experience replay, continuous-time control, and neural network approximation, enhancing robustness and efficiency in traffic regulation.

Findings

The proposed IRL method converges and stabilizes traffic dynamics.

Experience replay improves data efficiency and learning speed.

Numerical simulations demonstrate effectiveness in real-world scenarios.

Abstract

Existing data-driven and feedback traffic control strategies do not consider the heterogeneity of real-time data measurements. Besides, traditional reinforcement learning (RL) methods for traffic control usually converge slowly for lacking data efficiency. Moreover, conventional optimal perimeter control schemes require exact knowledge of the system dynamics and thus would be fragile to endogenous uncertainties. To handle these challenges, this work proposes an integral reinforcement learning (IRL) based approach to learning the macroscopic traffic dynamics for adaptive optimal perimeter control. This work makes the following primary contributions to the transportation literature: (a) A continuous-time control is developed with discrete gain updates to adapt to the discrete-time sensor data. (b) To reduce the sampling complexity and use the available data more efficiently, the experience replay (ER) technique is introduced to the IRL algorithm. (c) The proposed method relaxes the requirement on model calibration in a "model-free" manner that enables robustness against modeling uncertainty and enhances the real-time performance via a data-driven RL algorithm. (d) The convergence of the IRL-based algorithms and the stability of the controlled traffic dynamics are proven via the Lyapunov theory. The optimal control law is parameterized and then approximated by neural networks (NN), which moderates the computational complexity. Both state and input constraints are considered while no model linearization is required. Numerical examples and simulation experiments are presented to verify the effectiveness and efficiency of the proposed method.