Policy iteration: for want of recursive feasibility, all is not lost

TL;DR

This paper analyzes the recursive feasibility and stability of policy iteration (PI) for nonlinear systems, introduces modifications to ensure feasibility, and demonstrates near-optimal control with stability guarantees.

Contribution

It provides novel conditions for recursive robust stability of PI, introduces PI+ to guarantee recursive feasibility, and maintains near-optimality in nonlinear control.

Findings

PI can ensure recursive robust stability under certain conditions.

Modified PI+ guarantees recursive feasibility and stability.

PI+ maintains near-optimality similar to standard PI.

Abstract

This paper investigates recursive feasibility, recursive robust stability and near-optimality properties of policy iteration (PI). For this purpose, we consider deterministic nonlinear discrete-time systems whose inputs are generated by PI for undiscounted cost functions. We first assume that PI is recursively feasible, in the sense that the optimization problems solved at each iteration admit a solution. In this case, we provide novel conditions to establish recursive robust stability properties for a general attractor, meaning that the policies generated at each iteration ensure a robust \KL-stability property with respect to a general state measure. We then derive novel explicit bounds on the mismatch between the (suboptimal) value function returned by PI at each iteration and the optimal one. Afterwards, motivated by a counter-example that shows that PI may fail to be recursively feasible, we modify PI so that recursive feasibility is guaranteed a priori under mild conditions. This modified algorithm, called PI+, is shown to preserve the recursive robust stability when the attractor is compact. Additionally, PI+ enjoys the same near-optimality properties as its PI counterpart under the same assumptions. Therefore, PI+ is an attractive tool for generating near-optimal stabilizing control of deterministic discrete-time nonlinear systems.