Convergent NMPC-based Reinforcement Learning Using Deep Expected Sarsa and Nonlinear Temporal Difference Learning

TL;DR

This paper introduces a novel reinforcement learning approach integrating deep Expected Sarsa with nonlinear model predictive control, improving stability and reducing computational load for optimal control tasks.

Contribution

It proposes two methods combining RL with NMPC, enhancing stability, convergence, and real-time performance in nonlinear control systems.

Findings

Approach converges to a local optimum without instability.

Neural network reduces computational burden by half.

Method stabilizes learning in nonlinear environments.

Abstract

In this paper, we present a learning-based nonlinear model predictive controller (NMPC) using an original reinforcement learning (RL) method to learn the optimal weights of the NMPC scheme, for which two methods are proposed. Firstly, the controller is used as the current action-value function of a deep Expected Sarsa where the subsequent action-value function, usually obtained with a secondary NMPC, is approximated with a neural network (NN). With respect to existing methods, we add to the NN's input the current value of the NMPC's learned parameters so that the network is able to approximate the action-value function and stabilize the learning performance. Additionally, with the use of the NN, the real-time computational burden is approximately halved without affecting the closed-loop performance. Secondly, we combine gradient temporal difference methods with a parametrized NMPC as a function approximator of the Expected Sarsa RL method to overcome the potential parameters' divergence and instability issues when nonlinearities are present in the function approximation. The simulation results show that the proposed approach converges to a locally optimal solution without instability problems.