Safety Reinforced Model Predictive Control (SRMPC): Improving MPC with Reinforcement Learning for Motion Planning in Autonomous Driving

TL;DR

This paper introduces a novel approach combining safe reinforcement learning with model predictive control to enhance motion planning in autonomous driving, enabling exploration beyond local optima while ensuring safety.

Contribution

It proposes integrating constrained reinforcement learning with MPC using a safety index and learned Lagrangian multipliers to improve safety and solution quality in autonomous driving.

Findings

Outperforms traditional MPC in safety and performance

Enables exploration beyond local optima in motion planning

Demonstrates effectiveness in highway scenarios

Abstract

Model predictive control (MPC) is widely used for motion planning, particularly in autonomous driving. Real-time capability of the planner requires utilizing convex approximation of optimal control problems (OCPs) for the planner. However, such approximations confine the solution to a subspace, which might not contain the global optimum. To address this, we propose using safe reinforcement learning (SRL) to obtain a new and safe reference trajectory within MPC. By employing a learning-based approach, the MPC can explore solutions beyond the close neighborhood of the previous one, potentially finding global optima. We incorporate constrained reinforcement learning (CRL) to ensure safety in automated driving, using a handcrafted energy function-based safety index as the constraint objective to model safe and unsafe regions. Our approach utilizes a state-dependent Lagrangian multiplier, learned concurrently with the safe policy, to solve the CRL problem. Through experimentation in a highway scenario, we demonstrate the superiority of our approach over both MPC and SRL in terms of safety and performance measures.