Proximal Policy Optimization Learning based Control of Congested Freeway Traffic

TL;DR

This paper introduces a reinforcement learning-based control method using proximal policy optimization to stabilize congested freeway traffic, demonstrating improved performance and robustness over traditional control methods through numerical simulations.

Contribution

It develops a PPO-based delay-compensated feedback controller for traffic flow stabilization that learns control gains without explicit system dynamics knowledge.

Findings

PPO control converges faster with less effort in delay-free systems.

PPO performs comparably to backstepping control under input delay conditions.

PPO exhibits robustness to parameter perturbations, unlike backstepping control.

Abstract

This study proposes a delay-compensated feedback controller based on proximal policy optimization (PPO) reinforcement learning to stabilize traffic flow in the congested regime by manipulating the time-gap of adaptive cruise control-equipped (ACC-equipped) vehicles.The traffic dynamics on a freeway segment are governed by an Aw-Rascle-Zhang (ARZ) model, consisting of $2\times 2$ nonlinear first-order partial differential equations (PDEs).Inspired by the backstepping delay compensator [18] but different from whose complex segmented control scheme, the PPO control is composed of three feedbacks, namely the current traffic flow velocity, the current traffic flow density and previous one step control input. The control gains for the three feedbacks are learned from the interaction between the PPO and the numerical simulator of the traffic system without knowing the system dynamics. Numerical simulation experiments are designed to compare the Lyapunov control, the backstepping control and the PPO control. The results show that for a delay-free system, the PPO control has faster convergence rate and less control effort than the Lyapunov control. For a traffic system with input delay, the performance of the PPO controller is comparable to that of the Backstepping controller, even for the situation that the delay value does not match. However, the PPO is robust to parameter perturbations, while the Backstepping controller cannot stabilize a system where one of the parameters is disturbed by Gaussian noise.