Reinforcement Learning-based Control of Nonlinear Systems using Carleman Approximation: Structured and Unstructured Designs

TL;DR

This paper introduces a novel RL control approach for nonlinear systems using Carleman linearization, enabling real-time learning of controllers with stability guarantees and structured representations.

Contribution

It develops a Carleman-based RL control framework with finite-dimensional truncation, stability analysis, and structured/sparse controller designs, advancing data-driven nonlinear control.

Findings

Proposes a real-time RL control algorithm in Carleman space.

Provides stability conditions for truncated and structured controllers.

Demonstrates near-optimal performance through numerical examples.

Abstract

We develop data-driven reinforcement learning (RL) control designs for input-affine nonlinear systems. We use Carleman linearization to express the state-space representation of the nonlinear dynamical model in the Carleman space, and develop a real-time algorithm that can learn nonlinear state-feedback controllers using state and input measurements in the infinite-dimensional Carleman space. Thereafter, we study the practicality of having a finite-order truncation of the control signal, followed by its closed-loop stability analysis. Finally, we develop two additional designs that can learn structured as well as sparse representations of the RL-based nonlinear controller, and provide theoretical conditions for ensuring their closed-loop stability. We present numerical examples to show how our proposed method generates closed-loop responses that are close to the optimal performance of the nonlinear plant. We also compare our designs to other data-driven nonlinear RL control methods such as those based on neural networks, and illustrate their relative advantages and drawbacks.