Verification of Dissipativity and Evaluation of Storage Function in Economic Nonlinear MPC using Q-Learning

TL;DR

This paper introduces a Q-Learning based method to verify dissipativity and evaluate storage functions in economic nonlinear MPC, addressing the challenge of stability analysis in constrained nonlinear systems.

Contribution

It presents a novel approach using undiscounted Q-Learning to obtain storage functions for dissipative systems in the context of nonlinear MPC.

Findings

Q-Learning can effectively capture storage functions for dissipative systems.

The method is validated through multiple case studies.

It works with rich parameterizations in discrete-time constrained systems.

Abstract

In the Economic Nonlinear Model Predictive (ENMPC) context, closed-loop stability relates to the existence of a storage function satisfying a dissipation inequality. Finding the storage function is in general -- for nonlinear dynamics and cost -- challenging, and has attracted attentions recently. Q-Learning is a well-known Reinforcement Learning (RL) techniques that attempts to capture action-value functions based on the state-input transitions and stage cost of the system. In this paper, we present the use of the Q-Learning approach to obtain the storage function and verify the dissipativity for discrete-time systems subject to state-input constraints. We show that undiscounted Q-learning is able to capture the storage function for dissipative problems when the parameterization is rich enough. The efficiency of the proposed method will be illustrated in the different case studies.