Towards Safe Reinforcement Learning Using NMPC and Policy Gradients: Part II - Deterministic Case

TL;DR

This paper develops a method combining deterministic policy gradients with safety constraints via robust NMPC, focusing on linear models to ensure safety during learning, with potential extensions to nonlinear systems.

Contribution

It introduces a safe deterministic policy gradient approach using robust linear MPC to explicitly enforce safety constraints during policy learning.

Findings

Safe policy learning is feasible with robust linear MPC.

The method maintains safety throughout the learning process.

Extension to nonlinear MPC is possible but more complex.

Abstract

In this paper, we present a methodology to deploy the deterministic policy gradient method, using actor-critic techniques, when the optimal policy is approximated using a parametric optimization problem, where safety is enforced via hard constraints. For continuous input space, imposing safety restrictions on the exploration needed to deploying the deterministic policy gradient method poses some technical difficulties, which we address here. We will investigate in particular policy approximations based on robust Nonlinear Model Predictive Control (NMPC), where safety can be treated explicitly. For the sake of brevity, we will detail the construction of the safe scheme in the robust linear MPC context only. The extension to the nonlinear case is possible but more complex. We will additionally present a technique to maintain the system safety throughout the learning process in the context of robust linear MPC. This paper has a companion paper treating the stochastic policy gradient case.