Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control

TL;DR

This paper compares Deep Reinforcement Learning and Model Predictive Control for adaptive cruise control, showing DRL's comparable performance under ideal conditions and better robustness with modeling errors.

Contribution

It provides a direct comparison between DRL and MPC for ACC using a simplified model, highlighting DRL's robustness to modeling errors and its performance limits.

Findings

DRL matches MPC performance with long prediction horizon in ideal conditions.

DRL's episode cost is only 5.8% higher than the optimal IPO solution.

DRL outperforms MPC when there are significant modeling errors.

Abstract

This study compares Deep Reinforcement Learning (DRL) and Model Predictive Control (MPC) for Adaptive Cruise Control (ACC) design in car-following scenarios. A first-order system is used as the Control-Oriented Model (COM) to approximate the acceleration command dynamics of a vehicle. Based on the equations of the control system and the multi-objective cost function, we train a DRL policy using Deep Deterministic Policy Gradient (DDPG) and solve the MPC problem via Interior-Point Optimization (IPO). Simulation results for the episode costs show that, when there are no modeling errors and the testing inputs are within the training data range, the DRL solution is equivalent to MPC with a sufficiently long prediction horizon. Particularly, the DRL episode cost is only 5.8% higher than the benchmark solution provided by optimizing the entire episode via IPO. The DRL control performance degrades when the testing inputs are outside the training data range, indicating inadequate generalization. When there are modeling errors due to control delays, disturbances, and/or testing with a High-Fidelity Model (HFM) of the vehicle, the DRL-trained policy performs better with large modeling errors while having similar performance as MPC when the modeling errors are small.