End-to-End Reinforcement Learning of Koopman Models for Economic Nonlinear Model Predictive Control

TL;DR

This paper introduces an end-to-end reinforcement learning approach for Koopman surrogate models to improve economic nonlinear model predictive control, demonstrating superior performance and adaptability over traditional system identification and neural network methods.

Contribution

The paper presents a novel end-to-end reinforcement learning method for Koopman models, enhancing (e)NMPC performance and adaptability compared to existing training approaches.

Findings

End-to-end trained Koopman models outperform system identification models in (e)NMPC.

The approach enables (e)NMPC controllers to adapt to control setting changes without retraining.

Compared to neural network controllers, Koopman-based (e)NMPC shows better reaction to system variations.

Abstract

(Economic) nonlinear model predictive control ((e)NMPC) requires dynamic models that are sufficiently accurate and computationally tractable. Data-driven surrogate models for mechanistic models can reduce the computational burden of (e)NMPC; however, such models are typically trained by system identification for maximum prediction accuracy on simulation samples and perform suboptimally in (e)NMPC. We present a method for end-to-end reinforcement learning of Koopman surrogate models for optimal performance as part of (e)NMPC. We apply our method to two applications derived from an established nonlinear continuous stirred-tank reactor model. The controller performance is compared to that of (e)NMPCs utilizing models trained using system identification, and model-free neural network controllers trained using reinforcement learning. We show that the end-to-end trained models outperform those trained using system identification in (e)NMPC, and that, in contrast to the neural network controllers, the (e)NMPC controllers can react to changes in the control setting without retraining.