Unified Automatic Control of Vehicular Systems with Reinforcement Learning

TL;DR

This paper introduces a unified reinforcement learning approach for controlling mixed autonomy vehicular systems, achieving significant efficiency improvements and revealing emergent traffic management behaviors.

Contribution

It presents a streamlined multi-task RL methodology for diverse vehicular systems, enabling high-performance control strategies with minimal manual tuning.

Findings

Empirical improvements of 15-60% over human driving baseline.

Discovery of emergent behaviors like wave mitigation and ramp metering.

Validated control strategies align with emergent behaviors.

Abstract

Emerging vehicular systems with increasing proportions of automated components present opportunities for optimal control to mitigate congestion and increase efficiency. There has been a recent interest in applying deep reinforcement learning (DRL) to these nonlinear dynamical systems for the automatic design of effective control strategies. Despite conceptual advantages of DRL being model-free, studies typically nonetheless rely on training setups that are painstakingly specialized to specific vehicular systems. This is a key challenge to efficient analysis of diverse vehicular and mobility systems. To this end, this article contributes a streamlined methodology for vehicular microsimulation and discovers high performance control strategies with minimal manual design. A variable-agent, multi-task approach is presented for optimization of vehicular Partially Observed Markov Decision Processes. The methodology is experimentally validated on mixed autonomy traffic systems, where fractions of vehicles are automated; empirical improvement, typically 15-60% over a human driving baseline, is observed in all configurations of six diverse open or closed traffic systems. The study reveals numerous emergent behaviors resembling wave mitigation, traffic signaling, and ramp metering. Finally, the emergent behaviors are analyzed to produce interpretable control strategies, which are validated against the learned control strategies.