Incremental Policy Iteration for Unknown Nonlinear Systems with Stability and Performance Guarantees

TL;DR

This paper introduces an incremental policy iteration algorithm for unknown nonlinear systems that guarantees stability and near-optimal performance, using model-free adaptive dynamic programming with recursive least squares.

Contribution

It develops a novel incremental policy iteration method combining recursive least squares with ADP, enabling stable and near-optimal control without requiring a stabilizing initial policy.

Findings

The method guarantees closed-loop stability.

It achieves near-optimal policies through iterative learning.

Numerical simulations confirm effectiveness.

Abstract

This paper proposes a general incremental policy iteration adaptive dynamic programming (ADP) algorithm for model-free robust optimal control of unknown nonlinear systems. The approach integrates recursive least squares estimation with linear ADP principles, which greatly simplifies the implementation while preserving adaptive learning capabilities. In particular, we develop a sufficient condition for selecting a discount factor such that it allows learning the optimal policy starting with an initial policy that is not necessarily stabilizing. Moreover, we characterize the robust stability of the closed-loop system and the near-optimality of iterative policies. Finally, we perform numerical simulations to demonstrate the effectiveness of the proposed method.

Equations

$$x_{k+1}=f(x_k,u_k),\forall k\in {\mathbb{Z}}_{\ge 0},$$

$$J_{\gamma }(x,\mathbf{u})=$$

$$V_{\gamma }^{\star }(x):=\underset{\mathbf{u}}{\min }{J}_{\gamma }(x,\mathbf{u})<+\infty .$$

$$V_{\gamma }^{\star }(x)=\underset{u\in \mathcal{U}}{\min }\{\ell (x,u)+\gamma V_{\gamma }^{\star }(f(x,u))\},\forall x\in \Omega .$$

$$H_{\gamma }^{\star }(x):=\arg \underset{u\in \mathcal{U}}{\min }\{\ell (x,u)+\gamma V_{\gamma }^{\star }(f(x,u))\}.$$

$$\Delta x_{k+1}=A_{k-1}\Delta x_k+B_{k-1}\Delta u_k+O(\Delta x_k^2,\Delta u_k^2),$$

$$X_k:=\left[\begin{array}{c}\Delta x_k\\ \Delta u_k\end{array}\right],$$

$$\hat{\Theta}_{k-1}:=\big{[}\hat{A}_{k-1}\ \hat{B}_{k-1}\big{]}^{\top}$$

$$\Delta {\hat{x}}_{k+1}^{\top }=X_k^{\top }{\hat{\Theta }}_{k-1}.$$

$$\Delta {\hat{x}}_{k+1}={\hat{A}}_{k-1}\Delta x_k+{\hat{B}}_{k-1}\Delta u_k.$$

$$\displaystyle\ell\big{(}x,h_{\gamma}^{0}(x)\big{)}\!+\!\gamma W_{\gamma}^{0}\big{(}\hat{\phi}(1,x,h_{\gamma}^{0}(x))\big{)}\leq\bar{\alpha}_{V}(\sigma(x),\gamma),$$

$$\displaystyle\begin{aligned} \Gamma&\big{(}\hat{\phi}(1,x,h_{\gamma}\big{)}-\Gamma(x)\leq-\alpha_{\Gamma}(\sigma(x))+\ell\big{(}x,h_{\gamma}\big{)},\end{aligned}$$

$${\hat{V}}_{\gamma }^i(x)\ge \ell (x,h_{\gamma }^{i+1}(x))+\gamma W_{\gamma }^i(\hat{\varphi }(1,x,h_{\gamma }^{i+1}(x))),$$

$$x_{k+1}$$

$$=x_k+{\hat{A}}_{k-1}\Delta x_k+{\hat{B}}_{k-1}\Delta h_{\gamma ,k}^i+{\Delta }_{\mathrm{IME}},$$

$$|V_{\gamma }^{\star }(x_k)-{\hat{V}}_{\gamma }^{\star }(x_k)|\le \frac{\gamma L_{\ell }{\epsilon }_{\mathrm{IME}}}{1-\gamma },$$

$${\hat{V}}_{\gamma }^{\star }(x_0)-V_{\gamma }^{\star }(x_0)=$$

$${\hat{u}}_k^{\star }\in {\hat{H}}_{\gamma }^{\star }({\hat{x}}_{k-1})=\underset{{\hat{u}}_k\in \mathcal{U}(x_{k-1})}{\mathrm{argmin}}\{\ell ({\hat{x}}_{k-1},{\hat{u}}_{k-1})+{\hat{V}}_{\gamma }^{\star }({\hat{x}}_{k+1})\},$$

$${\hat{V}}_{\gamma }^{\star }(x_0)-V_{\gamma }^{\star }(x_0)\le$$

$$|{\hat{V}}_{\gamma }^{\star }(x_0)-V_{\gamma }^{\star }(x_0)|\le$$

$$\le$$

$$x_{k+1}\in x_k+{\hat{A}}_{k-1}\Delta x_k+{\hat{B}}_{k-1}\Delta h_{\gamma ,k}^i+\rho (x_k)\mathbb{B},$$

$$\ell (x_0,h_{\gamma }^{i+1}(x_0))+\gamma W_{\gamma }^i({\hat{x}}_1^{i+1})\le {\hat{V}}_{\gamma }^i(x_0),$$

$$\gamma \ell ({\hat{x}}_1^{i+1},h_{\gamma }^{i+1}({\hat{x}}_1))+{\gamma }^2{W}_{\gamma }^i({\hat{x}}_2^{i+1})\le \gamma {\hat{V}}^i({\hat{x}}_1).$$

$$\ell (x_0,h_{\gamma }^{i+1}(x_0))+\gamma \ell ({\hat{x}}_1,h_{\gamma }^{i+1}({\hat{x}}_1))+{\gamma }^2{W}_{\gamma }^i({\hat{x}}_2^{i+1})\le {\hat{V}}_{\gamma }^i(x_0).$$

$$\sum _{k=0}^{N-1}{\gamma }^k\ell ({\hat{x}}_k^{i+1},h_{\gamma }^{i+1}({\hat{x}}_k))+{\gamma }^N{W}_{\gamma }^i({\hat{x}}_N^{i+1})\le {\hat{V}}_{\gamma }^i(x_0),$$

$$\sigma ({\hat{\varphi }}^{\star }(k,x))\le {\beta }^{\star }(\sigma (x),k),\forall k\in {\mathbb{Z}}_{\ge 0}.$$

$$\ell (x,h_{\gamma }^{\star }(x))\le {\hat{V}}_{\gamma }^{\star }(x)\le {\hat{V}}_{\gamma }^0(x)\le {\bar{\alpha }}_V(\sigma (x)).$$

$${\hat{V}}_{\gamma }^{\star }(x)=\ell (x,h_{\gamma }^{\star }(x))+\gamma {\hat{V}}_{\gamma }^{\star }(v),$$

$${\hat{V}}_{\gamma }^{\star }(v)-{\hat{V}}_{\gamma }^{\star }(x)=-\frac{1}{\gamma }\ell (x,h_{\gamma }^{\star }(x))+\frac{1-\gamma }{\gamma }{\hat{V}}_{\gamma }^{\star }(x),$$

$${\hat{V}}_{\gamma }^{\star }(v)-{\hat{V}}_{\gamma }^{\star }(x)\le -\frac{1}{\gamma }\ell (x,h_{\gamma }^{\star }(x))+\frac{1-\gamma }{\gamma }{\bar{\alpha }}_V(\sigma (x)).$$

$${\alpha }_{\Gamma }(\sigma (x))\le {{\rm Y}}_{\gamma }^{\star }(x)\le {\bar{\alpha }}_V(x)+\frac{1}{{\gamma }^{\star }}{\bar{\alpha }}_W(\sigma (x)).$$

Full text

I Introduction

Adaptive Dynamic Programming (ADP) is a powerful method for solving the Hamilton-Jacobi-Bellman (HJB) equation in optimal control problems for uncertain and nonlinear systems [1, 2]. It is particularly effective in mitigating the curse of dimensionality by approximating the value function and control policy using function approximators, such as neural networks and polynomial functions. ADP can be categorized into two primary algorithmic frameworks: policy iteration (PI) and value iteration (VI). Both approaches typically rely on high-fidelity simulation models in offline training. However, obtaining such models can be challenging in practical applications involving highly nonlinear and uncertain physical systems [3].

For a linear time-invariant system, the system matrices are inherently embedded in the temporal evolution of input-output data, as they satisfy an overdetermined equation constraint. This allows the system model to be fully represented by data, enabling linear ADP methods to achieve optimal control without requiring an explicit system model [4, 5]. A fundamental question arises: Can nonlinear systems, like their linear counterparts, leverage data-driven strategies to achieve optimal control, thereby eliminating the need for explicit model identification or neural network model training? Addressing this challenge requires extending linear ADP methods to nonlinear system control. One promising approach is incremental control [6], which iteratively refines control strategies using locally linearized approximations. Building on this idea, recent research has explored a hybrid framework that integrates nonlinear incremental control techniques with linear VI-ADP [7]. While this method has demonstrated effectiveness in numerical simulations [8], its stability guarantees remain underdeveloped, highlighting the need for further theoretical analysis. The importance of stability-guaranteed design in uncertain systems has been widely recognized in adaptive control research. A representative work is [9], which demonstrates closed-loop stability and asymptotic tracking for uncertain nonlinear systems.

Although stability guarantees for standard ADP methods are well-established in both linear and nonlinear settings [11, 10, 12, 13], they can not directly apply to ADP algorithms with incremental models. The key challenge lies in the model approximation errors introduced by using an incremental linear model to represent nonlinear dynamics, which complicates convergence and stability analysis. Furthermore, since VI does not explicitly refine the control policy at each iteration, it lacks a direct mechanism to ensure improvement in stability during training, making VI less suitable for online learning [11]. Although PI can provide online learning solutions with stability guarantees, it requires an initially stabilizing policy, which is computationally expensive to obtain, especially when the system model is unknown [14]. Given these challenges, developing incremental PI-ADP algorithms without initially stabilizing policy requirement is of great significance for achieving online learning control with theoretical guarantees.

Motivated by the above discussions, this paper proposes an Incremental Policy Iteration (IPI) framework, which reformulates the integration of the incremental control technique and PI while eliminating the need for an initially stabilizing policy. Theoretical guarantees for IPI, including near-optimality and robust stability, are rigorously established. The main contributions of this work are threefold:

• i)

A general IPI algorithm is proposed, where a first-order Taylor series approximation model of the system dynamics is identified using recursive least squares (RLS) methods [15]. This facilitates next-step state computation during offline training while enabling online policy optimization with limited data, providing a model-free controller design adaptable to dynamic system variations.

• ii)

To handle errors from linearized approximations, system identification, and value-function surrogates, we design iteration rules that synchronize model updates, value approximations, and policy improvements. These rules bound the accumulated errors, ensuring that the generated policies remain provably near-optimal for nonlinear control using linear ADP methods.

• iii)

Using a general indicator function to define attractor-related stability, we show via Lyapunov analysis that the proposed IPI algorithm converges to a stabilizing policy, given a sufficient number of iterations. Furthermore, we establish an explicit relationship between closed-loop stability and the discount factor, relaxing the requirement for an initially stabilizing policy.

II Preliminaries

II-A Notations

Denote the set of real numbers by $\mathbb{R}$, the set of integers by $\mathbb{Z}$, and the set of $n$ dimensional real number vectors by $\mathbb{R}^{n}$. Denote a subset of $\mathcal{A}$ satisfying $(\cdot)$ by $\mathcal{A}_{(\cdot)}$. A function $\alpha:[0,a)\to[0,\infty)$ is of class $\mathcal{K}$ if $\alpha(0)=0$ and $\alpha(r)$ is continuous and strictly increasing; it is of class $\mathcal{K}_{\infty}$ if additionally $a=\infty$ and $\alpha(r)\to\infty$ as $r\to\infty$; a function $\beta:[0,\infty)\times[0,\infty)\to[0,\infty)$ is of class $\mathcal{KL}$ if for each fixed $t\geq 0$, $\beta(r,t)$ is of class $\mathcal{K}$, and for each fixed $r\geq 0$, $\beta(r,t)$ is continuous, strictly decreasing, and $\beta(r,t)\to 0$ as $t\to\infty$. The Euclidean norm of a vector $x\in\mathbb{R}^{n_{x}}$ with $n_{x}\in\mathbb{Z}_{>0}$ is denoted by $\|x\|$ and the distance of $x\in\mathbb{R}^{n_{x}}$ to a nonempty closed set $\mathcal{A}\subset\mathbb{R}^{n_{x}}$ is denoted by $\|x\|_{\mathcal{A}}:\inf\{\|x-y\|:y\in\mathcal{A}\}$. Given $\mathcal{A}$, the map $\sigma(x):\mathbb{R}^{n_{x}}\rightarrow\mathbb{R}_{\geq 0}$ is a proper indicator function of set $\mathcal{A}$ whenever $\sigma$ is continuous and there exist $\underline{\sigma},\bar{\sigma}\in\mathcal{K}_{\infty}$ such that $\underline{\sigma}(\|x\|_{\mathcal{A}})\leq\sigma(x)\leq\bar{\sigma}(\|x\|_{\mathcal{A}})$.

II-B Model and cost function

Consider a nonlinear system in the form of

[TABLE]

where $x_{k}:=x(t_{k})\in\Omega\subset\mathbb{R}^{n_{x}}$ is the state on the compact set $\Omega$, $u_{k}:=u(t_{k})\in\mathcal{U}(x_{k})\subseteq\mathbb{R}^{n_{u}}$ is the control input at time instant $t_{k}$ with $t_{k+1}=t_{k}+\Delta t$, $\Delta t>0$ is a fixed sampling time interval, $\mathcal{U}(x_{k})$ is a non-empty compact set of admissible inputs at state $x_{k}$, $k\in\mathbb{Z}_{\geq 0}$, and $n_{x},n_{u}\in\mathbb{Z}_{>0}$ are the dimensions of state and control input, respectively. The vector field $f(\cdot,\cdot):\mathbb{R}^{n_{x}}\times\mathbb{R}^{n_{u}}\rightarrow\mathbb{R}^{n_{x}}$ is unknown but assumed to be Jacobian-Lipschitz on $\Omega\times\mathcal{U}$

The solution to (1) is denoted by $\phi(k,x,{\mathbf{u}\mid_{k}})$ at time $t_{k}$ with the initial state $x$ and an admissible truncated control sequence ${\mathbf{u}\mid_{k}}:=\{u(0),\cdots,u(k-1)\}$. We use the convention $\phi(0,x,{\mathbf{u}\mid_{0}})=x$. We wish to find an infinite-length sequence of admissible inputs $\mathbf{u}$ by using the available data that minimizes the infinite horizon cost

[TABLE]

where $\gamma\in(0,1)$ is a cost discount factor, and $\ell:\mathbb{R}^{n_{x}}\times\mathbb{R}^{n_{u}}\rightarrow\mathbb{R}_{\geq 0}$ is a non-negative stage cost such that $|\ell(x,u)-\ell(y,u)|\leq L_{\ell}\|x-y\|$ with a known constant $L_{\ell}>0$, $x,y\in\mathbb{R}^{n_{x}}$. For any $x\in\Omega$, the optimal value function associated with the minimization of (2) is denoted by

[TABLE]

As a result, the Bellman equation becomes

[TABLE]

The optimal inputs for any state $x\in\Omega$ constitute a non-empty set as

[TABLE]

For the nonlinear system (1) with a general cost function (2), computing $H^{\star}_{\gamma}$ in (4) is extremely challenging, especially when the system dynamics are unknown. Therefore, it is necessary to utilize dynamic programming iterations to obtain the feedback law, ensuring that its cost asymptotically converges to the optimal value.

To achieve this, the following necessary assumptions are given [1]. The existence of the optimal and stabilizing control sequence, as defined below, is a prerequisite for optimization.

Assumption 1

For any $x\in\mathbb{R}^{n_{x}}$ and any $\gamma\in(0,1)$, there exists an optimal sequence of admissible inputs $\mathbf{u}^{\star}(x)$ such that $V^{\star}_{\gamma}(x)=J_{\gamma}(x,\mathbf{u}^{\star}(x))<\infty$ and for any infinite-length sequence of admissble inputs $\mathbf{u}$, $V_{\gamma}^{\star}(x)\leq J_{\gamma}(x,\mathbf{u})$. $\square$

Assumption 2

There exists $\bar{\alpha}_{V^{\star}}\in\mathcal{K}_{\infty}$ and $\gamma_{0}\in(0,1]$ such that for any $\gamma\in(0,\gamma_{0})$ and any $x\in\Omega$, $V^{\star}_{\gamma}(x)\leq\bar{\alpha}_{V^{\star}}(\sigma(x))$, where $V^{\star}_{\gamma}$ is given in (3). $\square$

III Problem Formulation

III-A Incremental policy iteration

This subsection introduces the incremental policy iteration, as shown in Algorithm 1, to iteratively obtain feedback laws.

Specifically, taking the Taylor expansion of (1) at state $x_{k}$, the following incremental model is obtained

[TABLE]

where the incremental state and control at time $k$ are defined as $\Delta x_{k}:=x_{k}-x_{k-1}$ and $\Delta u_{k}=u_{k}-u_{k-1}$, matrices $A_{k-1}:=\frac{\partial f}{\partial x}\mid_{x=x_{k-1}}$ and $B_{k-1}:=\frac{\partial f}{\partial u}\mid_{u=u_{k-1}}$ are partial derivatives of the dynamics with respect to the state and control at time $t_{k-1}$, and $O(\cdot)$ denotes the high-order remainder. Since $f$ is Jacobian-Lipschitz, the $O(\cdot)$ is bounded on $\Omega\times\mathcal{U}$. The nonlinear system (1) can be represented as this time-varying incremental model, in which $A_{k-1},B_{k-1}$ are expected to be identified by using RLS methods [15].

The concrete RLS identification principle is given below. With the following augmented system state

[TABLE]

the augmented system matrices

[TABLE]

are responsible for the one-step prediction as

[TABLE]

The identification is achieved by using ($\mathrm{LS.1}$)-($\mathrm{LS.4}$) in Algorithm 1 with a recursive manner. For the physical system matrices $\Theta_{k},k\in\mathbb{Z}_{\geq 0}$, denote the estimation error by $\tilde{\Theta}_{k}:=\hat{\Theta}_{k}-\Theta_{k}$. Define the system incremental error as $\Delta\Theta_{k}:=\Theta_{k}-\Theta_{k-1}$, which is determined by the dynamics (1). Based on the form of incremental model (5), it follows that there exists an upper bound $\varepsilon_{\Delta\Theta}$ such that $\|\Delta\Theta_{k}\|\leq\varepsilon_{\Delta\Theta}$, for all $k\in\mathbb{Z}_{>0}$. As a result, the estimation error is bounded and satisfies the following properties.

Lemma 1** ([15])**

For system (1), denote the changing of incremental model (5) upper bound by $\varepsilon_{\Delta\Theta}>0$. Then, when identifying $\Theta_{k}$ using ($\mathrm{LS.1}$)-($\mathrm{LS.4}$), there exists $\beta_{\Theta}\in\mathcal{K}\mathcal{L}$ such that $\tilde{\Theta}_{k}\leq\beta_{\Theta}(\varepsilon_{\Delta\Theta},k)$, $k\in\mathbb{Z}_{>0}$. $\square$

With the estimated matrices $\hat{\Theta}_{k}$ at time $t_{k}$, the estimated incremental state at successor time $t_{k+1}$ is given by

[TABLE]

A parameterized value function approximator $W^{i}_{\gamma}(\cdot)$ is introduced and the approximation value $\hat{V}_{\gamma}(\hat{x}_{k+1})$ at time $k+1$ is given by substituting $\hat{\phi}(1,x_{k},u_{k}):=\hat{x}_{k+1}=x_{k}+\Delta\hat{x}_{k+1}$ into the approximator $W^{i}_{\gamma}(\cdot)$, yielding ($\mathrm{PI.4}$).

The IPI is assumed to satisfy the following conditions.

Assumption 3

There exist a constant $0<\gamma_{0}\leq 1$, functions $\bar{\alpha}_{V}(\cdot,\gamma),\alpha_{\Gamma}(\cdot)\in\mathcal{K}_{\infty}$, a continuous function $\Gamma:\mathbb{R}^{x_{n}}\rightarrow\mathbb{R}_{\geq 0}$, such that $\forall\gamma\in(0,\gamma_{0}]$,

[TABLE]

for all $i\in\mathbb{Z}_{\geq 0}$, $x\in\Omega$. $\square$

Remark 1

Condition (7a) establishes the relationship between the stability (the distance of state $x$ from the attractor) of the original system (1) and the initial estimated value function $\hat{V}^{0}_{\gamma}$. There exists no assumption of stabilizing initial policy but only the bounded initial estimated value function by $\bar{\alpha}_{V}$ is required. This is reasonable and easy to implement because, for any given state $x\in\mathbb{R}^{n_{x}}$ and an initial policy $h_{\gamma}^{0}\in\mathcal{U}(x)$, $\sigma(x)$ and $\ell(x,h_{\gamma}^{0}(x))$ can be computed explicitly. By choosing an appropriate $\gamma$ and an approximation operator $W_{\gamma}^{0}(\cdot)$, condition (7a) can be satisfied. In particular, the smaller the $\gamma$, the easier it is to satisfy this condition, which can be reflected in selecting a sufficiently small $\gamma_{0}$. An appropriate form of the initial policy $h^{0}$ can be selected to satisfy $\ell(\hat{\phi}(k,x,h^{0}),h^{0}(\phi(k,x,h^{0})))\leq Ma^{k}\chi(\sigma(x))$ with $M,a>0$ and $\chi\in\mathcal{K}_{\infty}$, which may be not stabilizing when $a$ is strictly bigger than 1 [16].

Condition (7b) specifies the detectability property of system (1) with incremental model approximation (5), when consider $\ell$ as an output. This is natural as this shows the fact that by minimizing $\ell$ along the solution to (5), desirable stability properties should follow.

Condition (7c) indicates that the cost generated by the next policy $h_{\gamma}^{i+1}$, along with the discounted future cost, should not exceed the current estimated total cost. That is, each updated policy $h_{\gamma}^{i+1}$ is better than or at least equivalent to the previous policy $h_{\gamma}^{i}$. This is consistent with the upper bound form of the Bellman equation and ensures stable policy improvement. $\square$

With the IPI, the closed-loop system with approximation optimal controller is given by

[TABLE]

where $\Delta h^{i}_{\gamma,k}\in\Delta H_{\gamma}^{i}(x_{k})$ defined in ($\mathrm{PI.5}$) and $\Delta_{\mathrm{IME}}:=(\hat{A}_{k-1}-A_{k-1})\Delta x_{k}+(B_{k-1}-\hat{B}_{k-1})\Delta u_{k}+O(\Delta x_{k}^{2},\Delta(u^{i}_{k})^{2})$ is the total error of using IPI. From Lemma 1 and the boundedness of $O(\cdot)$, it is reasonable to assume that there exists a constant $\varepsilon_{\mathrm{IME}}>0$ such that $\|\Delta_{\mathrm{IME}}\|\leq\varepsilon_{\mathrm{IME}}$. Therefore, the upper bound of the error between the optimal value function $\hat{V}^{\star}_{\gamma}$ of the incremental model and that of the original nonlinear system (1) is deduced below.

Proposition 1

Consider the system (1) with the incremental approximation (8) controlled by policies generated from Algorithm 1. If there exists constant $\varepsilon_{\mathrm{IME}}>0$ such that $\|\Delta_{\mathrm{IME}}\|\leq\varepsilon_{\mathrm{IME}}$, then for all $k\in\mathbb{Z}_{\geq 0}$ and $\gamma\in(0,1)$,

[TABLE]

where $L_{\ell}$ is the Lipscitz constant of $\ell(\cdot,\cdot)$. $\square$

Proof. For any initial state $x_{0}\in\mathbb{R}^{n_{x}}$, $\gamma\in(0,1)$, $i\in\mathbb{Z}_{\geq 0}$, solution $x_{k+1}$ to (8) and solution $\hat{x}_{k+1}$ to (PI.3), let $u^{\star}_{k}\in H^{\star}_{\gamma}$ and $\hat{u}^{\star}_{k}\in\hat{H}^{\star}_{\gamma}$. By Bellman equation and the definition of $V_{\gamma}^{\star}$,

[TABLE]

As the following equation holds,

[TABLE]

we have that for all $\hat{u}_{k}\in\mathcal{U}(\hat{x}_{k-1})$, $k\in\mathbb{Z}_{>0}$,

[TABLE]

When selecting $\hat{u}_{k}=u^{\star}_{k}$, it follows that

[TABLE]

This completes the proof. $\blacksquare$

Denote $\rho(x_{k}):=\|\Delta_{\mathrm{IME}}\|$, which inspires the following perturbation set-valued closed-loop system

[TABLE]

where $\mathbb{B}$ is the unit closed ball of $\mathbb{R}^{n_{x}}$ centered at the origin, and $\Delta h^{i}_{\gamma,k}\in\Delta H^{i}_{\gamma}$ is defined in (PI.5).

III-B Study objectives

The definitions of desired properties for IPI are given.

Definition 1** (Near-optimality,[14])**

Consider system (1) with the infinite horizon cost (2) and the minimization value $V_{\gamma}^{\star}$. A policy iteration algorithm is with near-optimality if there exists a bound $\varepsilon_{V^{\star}}$ such that $\hat{V}^{i}_{\gamma}(x)-V^{\star}_{\gamma}(x)\leq\varepsilon_{V^{\star}}$, $\forall i\in\mathbb{Z}_{\geq 0}$ and $x\in\Omega$. $\square$

Definition 2** (Robust stability,[14])**

The system (8) with the policy $h_{\gamma}^{i}$, $i\in\mathbb{Z}_{\geq 0}$, is robustly stable if there exists $\beta\in\mathcal{KL}$ and any given bound $\bar{\delta}\geq 0$ such that $\sigma(\phi(k,x,h_{\gamma}^{i}))\leq\max\{\beta(\sigma(x),k),\bar{\delta}\}$ for every $x\in\Omega$ and $k\in\mathbb{Z}_{\geq 0}$. $\square$

Since it is impractical to implement Algorithm 1 with infinite iterations and the approximation errors introduced by the incremental model, RLS, and the approximator cannot be ignored, our objectives are to

• i)

verify that it provides an approximate optimality guarantee, where the value error is bounded,

• ii)

and establish conditions under which the IPI can produce a robustly stable control policy within a finite number of iterations.

IV Main Results

IV-A Near-optimality

We first give the result on the improvement property of the IPI with respect to the value function.

Proposition 2

Under Assumption 3, for any $x\in\Omega$, $\gamma\in(0,\gamma_{0})$, $\hat{V}_{\gamma}^{i+1}(x)\leq\hat{V}_{\gamma}^{i}(x)$, $i\in\mathbb{Z}_{\geq 0}$. $\square$

Proof. Evaluating (7c) at $x_{0}\in\mathbb{R}^{n_{x}}$, one has

[TABLE]

where $\hat{x}^{i+1}_{k+1}:=\hat{\phi}(1,x_{k},h_{\gamma}^{i+1})$, $k\in\mathbb{Z}_{\geq 0}$. Also, since $\gamma\neq 0$, evaluating (7c) at $\hat{x}_{1}=\hat{\phi}(1,x_{0},h_{\gamma}^{i+1}(x_{0}))\in\mathbb{R}^{n_{x}}$ leads to

[TABLE]

Using (12) in (11) yields

[TABLE]

Repeating this process for $N-2$ more times leads to

[TABLE]

where $\hat{x}^{i+1}_{0}=x_{0}$. Letting $N\rightarrow\infty$ and given $\hat{V}^{i}_{\gamma}(x_{0})\geq 0$, $\forall x_{0}\in\mathbb{R}^{n_{x}}$, which hence can be dropped from the left hand side, inequality (13) leads to $\hat{V}_{\gamma}^{i+1}(x)\leq\hat{V}_{\gamma}^{i}(x),\forall x\in\Omega$. $\blacksquare$

The next proposition shows that the optimal policy for the incremental model verifies a $\mathcal{KL}$-stability property with respect to $\sigma$.

Proposition 3

If there exists $0<\gamma^{\star}\leq 1$ such that $(1-\gamma^{\star})\bar{\alpha}_{V}(s)\leq\alpha_{\Gamma}(s),\forall s\in\mathbb{R}_{>0}$, then for any $\gamma\in(\gamma^{\star},\gamma_{0})$, system (10) with optimal policies $h^{\star}_{\gamma}$ is $\mathcal{KL}$ stable with respect to $\sigma$, i.e., there exists $\beta^{\star}\in\mathcal{KL}$ such that for any $x\in\Omega$, any solution $\hat{\phi}^{\star}(\cdot,x)$ to (10) satisfies

[TABLE]

$\square$**

Proof. Let $\gamma\in(\gamma^{\star},\gamma_{0})$, $x\in\Omega$, and $v=\hat{\phi}(1,x,h^{\star}_{\gamma}(x))$ with $h^{\star}_{\gamma}(x)\in H^{\star}_{\gamma}(x)$. Since $\ell$ is non-negative and by using (7a), it follows that

[TABLE]

By definition of $\hat{V}^{\star}_{\gamma}$, one has that

[TABLE]

therefore

[TABLE]

which gives that

[TABLE]

Define $\Upsilon_{\gamma}^{\star}:=\hat{V}^{\star}_{\gamma}+\frac{1}{\gamma}\Gamma$. Combining (7b) with (14) yields the following bounds

[TABLE]

Denote $\bar{\alpha}_{\Upsilon}:=\bar{\alpha}_{V}+\frac{1}{\gamma^{\star}}\bar{\alpha}_{W}$ and $\underline{\alpha}_{\Upsilon}:=\alpha_{\Gamma}$. Moreover, from (14), it follows that

[TABLE]

Since $(1-\gamma^{\star})\bar{\alpha}_{V}(s)\leq\alpha_{\Gamma}(s),\forall s\in\mathbb{R}_{>0}$ and $\gamma\geq\gamma^{\star}$, we have that

[TABLE]

where $\alpha_{\Upsilon}(\cdot,\gamma):=\frac{\gamma-\gamma^{\star}}{1-\gamma^{\star}}\alpha_{\Gamma}(\cdot)\in\mathcal{K}_{\infty}$. By induction, it can be seen that there exists $\beta^{\star}\in\mathcal{KL}$, such that

[TABLE]

with $\beta^{\star}(s,k)\mapsto\underline{\alpha}_{\Upsilon}^{-1}(\{\min(\bar{\alpha}_{\Upsilon}(s),\gamma)\}^{k},k)$. This completes the proof. $\blacksquare$

We are now give the results about the near optimality.

Theorem 1

For any $x\in\Omega$, $i\in\mathbb{Z}_{\geq 0}$, $\gamma\in(\gamma^{\star},\gamma_{0})$, and any solution to system (8),

[TABLE]

with $\beta^{\star}\in\mathcal{K}\mathcal{L}$ form Proposition 3 and $\bar{\alpha}_{V}$ from Assumption 3. $\square$

Proof. Let $x\in\Omega$, $i\in\mathbb{Z}_{>0}$, $h^{i}_{\gamma}\in H^{i}_{\gamma},h^{\star}_{\gamma}\in H^{\star}_{\gamma}$ and $\gamma\in(\gamma^{\star},\gamma_{0})$. By Bellman equation and the definition of $\hat{V}^{i}_{\gamma}(x)$,

[TABLE]

which is deduced by using (9). Since

[TABLE]

it follows that, $\forall u\in\mathcal{U}$,

[TABLE]

Therefore, by taking $u=\hat{h}^{\star}_{\gamma}\in\hat{H}^{\star}_{\gamma}(x)$ it follows that

[TABLE]

Using Proposition 2 and repeating the above reasoning $i-1$ times, we obtain

[TABLE]

Since $V^{\star}_{\gamma}\geq 0$, by Assumption 3, we have

[TABLE]

Combining (17), (18) with (16), Proposition 3, and noticing that $\bar{\alpha}_{V}$ is non-decreasing, we finally have (15). $\blacksquare$

IV-B Robust stability

Before giving the robust stability results, we establish the following Lyapunov property for the system during the policy iteration process.

Proposition 4

There exist $\underline{\alpha}_{Y}\in\mathcal{K}_{\infty},\bar{\alpha}_{Y},\alpha_{Y}:\mathbb{R}_{\geq 0}\times(\gamma^{\star},\gamma_{0})\rightarrow\mathbb{R}_{\geq 0}$ of class $\mathcal{K}_{\infty}$ in their first argument such that for any $i\in\mathbb{Z}_{\geq 0}$ there exist $Y^{i}_{\gamma}:\mathbb{R}^{n_{x}}\rightarrow\mathbb{R}_{\geq 0}$ satisfying

• (i)

For any $x_{k}\in\mathbb{R}^{n_{x}}$, $\underline{\alpha}_{Y}(\sigma(x_{k}))\leq Y^{i}_{\gamma}(x_{k})\leq\bar{\alpha}_{Y}(\sigma(x_{k}),\gamma)$;

• (ii)

For any $x_{k}\in\mathbb{R}^{n_{x}}$, Y^{i}_{\gamma}(\hat{x}_{k+1})-Y^{i}_{\gamma}({x}_{k})\leq\frac{1}{\gamma}\big{(}-\alpha_{Y}(\sigma(x_{k}),\gamma)+\Upsilon^{i}(\sigma(x_{k}),\gamma)\big{)},

for any $\gamma\in(\gamma^{\star},\gamma_{0})$, where $\Upsilon^{i}:\mathbb{R}_{\geq 0}\times(\gamma^{\star},\gamma_{0})\rightarrow\mathbb{R}_{\geq 0}$ is of class $\mathcal{K}_{\infty}$ in its first argument defined as $\Upsilon^{i}(\sigma,\gamma):=(1-\gamma)\gamma^{i}\bar{\alpha}_{V}(\beta^{\star}_{\gamma}(\sigma,i))$. $\square$

Proof. Since the stage cost $\ell$ is non-negative, for all $x_{k}\in\mathbb{R}^{n_{x}}$, one has that

[TABLE]

Moreover, according to (PI.3)-(PI.6) in Algorithm 1, $\hat{V}_{\gamma}^{i}(x_{k})=W_{\gamma}^{i}(x_{k})$, which yields that

[TABLE]

Defining $Y^{i}_{\gamma}:=\hat{V}^{i}_{\gamma}+\frac{1}{\gamma}\Gamma$, it follows from (7b) that

[TABLE]

By using Assumptions 3 and Proposition 2, one has

[TABLE]

Therefore, item (i) is proven.

From (PI.3)-(PI.6), it follows that

[TABLE]

In view of (7b) and (20), we have

[TABLE]

By using Theorem 1 and Proposition 3,

[TABLE]

Since $(1-\gamma^{\star})\bar{\alpha}_{V^{\star}}(s)\leq\alpha_{\Gamma}(s),\forall s\in\mathbb{R}_{>0}$ as stated in Proposition 3, the last two terms in (21) satisfy

[TABLE]

Define $\Upsilon^{i}(\sigma,\gamma):=(1-\gamma)\gamma^{i}\bar{\alpha}_{V}(\beta^{\star}_{\gamma}(\sigma,i))$. Item (ii) can be deduced and this completes the proof. $\blacksquare$

According to the form of $\Upsilon_{\gamma}^{i}$, it follows that item (ii) in Proposition 4 is a dissipative inequality of system (10) for which the supply rate consists of a negative term, namely $-\frac{1}{\gamma}\alpha_{Y}(\cdot,\gamma)$, and a non-negative term $\frac{1}{\gamma}\Upsilon^{i}(\cdot,\gamma)$ that can be made as small as desired by increasing $i$. Then, the following robust stability result is derived.

Theorem 2

Use the Lyapunov function definition and notation from Proposition 4. For any $x\in\Omega$, given $\delta\geq 0$ and $\tilde{\delta}:=\underline{\alpha}_{Y}(\delta)>0$, when Assumptions 1-3 hold, there exists $i^{\star}\in\mathbb{Z}_{\geq 0}$, such that

[TABLE]

for any $i\geq i^{\star}$, system (8) is robustly $\mathcal{KL}$-stable. $\square$

Proof. Denote \Delta:=\max\limits_{\rho\in\varepsilon_{\mathrm{IME}}\mathbb{B}}\big{(}\sigma(x+\rho)-\sigma(x)\big{)}\geq\sigma(\hat{\phi}(1,x,h^{i}_{\gamma}(x)))-\sigma(\phi(1,x,h^{i}_{\gamma}(x))), define $\tilde{\Delta}_{\gamma}:=\bar{\alpha}_{Y}(\Delta,\gamma)>0$. Using item (ii) in Proposition 4, define $v=\hat{\phi}(1,x,h^{i}_{\gamma}(x))$, one has that

[TABLE]

As $\Upsilon^{i}(\cdot,\gamma)$ is non-decreasing and $\alpha_{Y}(\cdot,\gamma)\in\mathcal{K}_{\infty}$, using item (i) of Proposition 4 and the fact that $Y^{i}_{\gamma}(x)\leq\tilde{\Delta}_{\gamma}$, (23) yields

[TABLE]

As $\beta^{\star}\in\mathcal{KL}$, for any $s\in\mathbb{R}_{\geq 0}$ and $i\in\mathbb{Z}_{\geq 0}$, it follows that

[TABLE]

As a result, when selecting $i^{\star}$ satisfying (22), it follows that

[TABLE]

Consequently, for any $i\geq i^{\star}$, when $Y_{\gamma}^{i}\geq\tilde{\delta}$,

[TABLE]

Therefore, there exists $\tilde{\beta}\in\mathcal{KL}$ such that for any solution $\hat{\phi}$ with respect to the incremental approximation model initialized at arbitrary $x$ and any $k\in\mathbb{Z}_{\geq 0}$,

[TABLE]

Therefore, by using item (i) in Proposition 4 and the definition of $\tilde{\delta}$, it follows that

[TABLE]

where $\beta(s,k):=\underline{\alpha}_{Y}^{-1}(\tilde{\beta}(\bar{\alpha}_{Y}(s,\gamma),k))$. This concludes the proof. $\blacksquare$

Remark 2

Theorem 2 shows that the proposed IPI framework guarantees robust $\mathcal{KL}$-stability, even without an initially stabilizing policy. This result is significant as it ensures that iterative policy updates naturally lead to stability despite model uncertainties and approximation errors. It also highlights the trade-off between convergence and robustness, as a smaller discount factor $\gamma$ accelerates stability but may slow policy improvement, while a larger $\gamma$ speeds up optimality convergence but requires more iterations for stability. This theorem strengthens IPI’s applicability in model-free control by ensuring stability in unknown nonlinear systems, making it a reliable approach for adaptive optimal control. $\square$

V Simulations

Consider a nominal nonlinear system (Model A) govern by the followig dynamics

[TABLE]

where $x_{k}=[x_{1,k};x_{2,k}]\in\mathbb{R}^{2},k\in\mathbb{Z}_{\geq 0}$. Select positive definite matrix $Q\in\mathbb{R}^{2\times 2}$ and positive constant $R>0$. The objective is to use the proposed IPI to steer the system from a given initial state $x_{0}$ to the equilibrium $(0,0)$, while minimizing the following performance index

[TABLE]

under the condition of knowing only the system input $u$ and state $x$ at the current and previous time steps $t_{k}$ and $t_{k-1}$.

Here, we consider the value approximator $W^{i}_{\gamma}(x_{k})$ as $x_{k}^{\top}P^{(i)}_{\gamma}x_{k}$, where $P^{(i)}_{\gamma}\in\mathbb{R}^{2\times 2}$ is a positive definite matrix to be found recursively. In this case, from (PI.1)-(PI.6), one has that

[TABLE]

Therefore, we can conclude that the policy is in the feedback form of system variables $(u_{k-1},x_{k},\Delta x_{k})$, and the gains are function of the current incremental model $(\hat{A}_{k-1},\hat{B}_{k-1})$.

For simplicity, select $Q=I_{2}$ and $R=1$. The implement procedure is illustrated as follows.

Offline training: Using (25) (Model A), a set of data $(\mathbf{x}_{0:N},\mathbf{u}_{0:N})$ is collected by applying a randomly generated input signal (a sinusoidal signal is considered in this paper). The corresponding $\hat{A}_{k-1},\hat{B}_{k-1}$ are identified by using batch LS for offline policy iteration. To verify Theorems 1-2, during the offline training, choose an initial policy satisfying (7a), $u^{0}_{k}=[-2.5\ -1]x_{k}$ and $\gamma=0.7$. It can be seen from the upper subfigure in Fig. 1 that the initial policy $u^{0}_{\gamma}$ is an unstable policy. $P^{i}_{\gamma}$ is updated by solving

[TABLE]

which implies that there exists $\alpha_{\Gamma}$ satisfying $\eqref{eq2 Assumption for IPI}$. Moreover, since $\Delta u^{i}_{k}$ is the analytic solution of (PI.5), it always makes (7c) hold.

Online implementation: To verify the robustness of the proposed method, we consider a different physical system (Model B) to control:

[TABLE]

where $u_{d,k}$ is the disturbance in the form of

[TABLE]

with Gaussian noise $w(k)$. The online iteration is implemented in a recursive manner with the offline trained policy as a baseline policy. That is, the kernel matrix $P$ is updated for each time step $t_{k}$:

[TABLE]

The incremental policy $\Delta u_{k}$ is improved based on the new $P_{k}$ by using (27). The state and incremental control response curves are shown in the second and third subfigures in Fig. 1. Thus, the trained policy makes the system stable while rejecting the uncertainty and disturbances brought by Model B.

Comparison: To further demonstrate the robustness of the proposed IPI approach, we compare it with the traditional nonlinear PI-ADP method [12]. In the conventional approach, a model neural network is first trained to approximate the system dynamics, and then this learned model is used to train the actor and critic networks via policy iteration with an initial stable policy. However, this indirect learning process introduces modeling errors that propagate through the control design, leading to degraded performance under uncertainties and disturbances when implementing online.

As both methods are trained using data collected from Model A, their ability to generalize to Model B under uncertainties and disturbances is evaluated. Fig. 1 demonstrates that the proposed IPI approach remains robust when applied to Model B, effectively handling disturbances and model uncertainties. In contrast, Fig. 2 shows that the traditional PI-ADP method, despite being trained on Model A, fails to maintain stability when uncertainties and disturbances are introduced. This degradation highlights the limitations of relying on a pre-trained model for control design, since inaccuracies in the learned dynamics can negatively impact policy performance. The IPI approach mitigates these issues by continuously adapting its policy, ensuring superior robustness and stability across different operating conditions.

VI Conclusions

In this paper, a general model-free incremental policy iteration framework for nonlinear systems is proposed, employing the recursive least squares method to identify linear approximation system matrices. This allows the offline pre-trained policy to be updated online with limited data. The approach avoids the high training cost and poor interpretability associated with the global approximation used in traditional nonlinear ADP methods, while robustly adapting to dynamic system variations through an incremental update mechanism. The near-optimality and robust stability of the algorithm are theoretically proven, providing a solid theoretical foundation. Future work will focus on extending the method to continuous systems and its applications in engineering practice.