A Theoretical Difficulty in Approximate Dynamic Programming with Input Constraints

TL;DR

This paper investigates a fundamental theoretical challenge in approximate dynamic programming with input constraints, highlighting issues in policy iteration convergence due to trajectory evaluation problems.

Contribution

It identifies a key theoretical difficulty in the convergence of constrained ADP policy iteration algorithms caused by trajectory evaluation issues.

Findings

Convergence issues arise in constrained ADP policy iteration.

The difficulty is linked to evaluating the same trajectory after policy updates.

This problem impacts the reliability of ADP in systems with actuator constraints.

Abstract

Equipping approximate dynamic programming (ADP) with inputconstraints has a tremendous significance. This enables ADP to be applied tothe systems with actuator limitations, which is quite common for dynamicalsystems. In a conventional constrained ADP framework, the optimal control issearched via a policy iteration algorithm, where the value under a constrainedcontrol is solved from a Hamilton-Jacobi-Bellman (HJB) equation while theconstrained control policy is improved based on the current estimated value.This concise and applicable method has been widely-used. However, the con-vergence of the existing policy iteration algorithm may possesses a theoreticaldifficulty, which might be caused by forcibly evaluating the same trajectoryeven though the control policy has already changed. This problem will beexplored in this paper.