Centralized Cooperation for Connected and Automated Vehicles at Intersections by Proximal Policy Optimization

TL;DR

This paper introduces a reinforcement learning-based centralized coordination scheme for connected vehicles at intersections, significantly reducing computation time and increasing traffic efficiency compared to traditional methods.

Contribution

It proposes a novel RL training algorithm, MA-PPO, and formulates intersection coordination as an RL problem, demonstrating superior performance over MPC in simulations.

Findings

MA-PPO accelerates learning with higher sample efficiency.

The proposed method reduces computation time to 1/400 of MPC.

Traffic efficiency increases by 4.5 times.

Abstract

Connected vehicles will change the modes of future transportation management and organization, especially at an intersection without traffic light. Centralized coordination methods globally coordinate vehicles approaching the intersection from all sections by considering their states altogether. However, they need substantial computation resources since they own a centralized controller to optimize the trajectories for all approaching vehicles in real-time. In this paper, we propose a centralized coordination scheme of automated vehicles at an intersection without traffic signals using reinforcement learning (RL) to address low computation efficiency suffered by current centralized coordination methods. We first propose an RL training algorithm, model accelerated proximal policy optimization (MA-PPO), which incorporates a prior model into proximal policy optimization (PPO) algorithm to accelerate the learning process in terms of sample efficiency. Then we present the design of state, action and reward to formulate centralized coordination as an RL problem. Finally, we train a coordinate policy in a simulation setting and compare computing time and traffic efficiency with a coordination scheme based on model predictive control (MPC) method. Results show that our method spends only 1/400 of the computing time of MPC and increase the efficiency of the intersection by 4.5 times.