# Incremental Policy Iteration for Unknown Nonlinear Systems with Stability and Performance Guarantees

## TL;DR

This paper introduces an incremental policy iteration algorithm for unknown nonlinear systems that guarantees stability and near-optimal performance, using model-free adaptive dynamic programming with recursive least squares.

## Contribution

It develops a novel incremental policy iteration method combining recursive least squares with ADP, enabling stable and near-optimal control without requiring a stabilizing initial policy.

## Key findings

- The method guarantees closed-loop stability.
- It achieves near-optimal policies through iterative learning.
- Numerical simulations confirm effectiveness.

## Abstract

This paper proposes a general incremental policy iteration adaptive dynamic programming (ADP) algorithm for model-free robust optimal control of unknown nonlinear systems. The approach integrates recursive least squares estimation with linear ADP principles, which greatly simplifies the implementation while preserving adaptive learning capabilities. In particular, we develop a sufficient condition for selecting a discount factor such that it allows learning the optimal policy starting with an initial policy that is not necessarily stabilizing. Moreover, we characterize the robust stability of the closed-loop system and the near-optimality of iterative policies. Finally, we perform numerical simulations to demonstrate the effectiveness of the proposed method.