Constrained Optimal Tracking Control of Unknown Systems: A Multi-Step Linear Programming Approach

TL;DR

This paper introduces a novel multi-step Q-learning and linear programming method for optimal tracking control of unknown systems with input constraints, improving convergence and eliminating the need for initial stabilizing policies.

Contribution

It develops a two-step transformation and a multi-step VI algorithm that handles input constraints and general stage costs without system model knowledge.

Findings

Enhanced convergence speed of the control algorithm

Elimination of initial stabilizing policy requirement

Successful simulation validation of the approach

Abstract

We study the problem of optimal state-feedback tracking control for unknown discrete-time deterministic systems with input constraints. To handle input constraints, state-of-art methods utilize a certain nonquadratic stage cost function, which is sometimes limiting real systems. Furthermore, it is well known that Policy Iteration (PI) and Value Iteration (VI), two widely used algorithms in data-driven control, offer complementary strengths and weaknesses. In this work, a two-step transformation is employed, which converts the constrained-input optimal tracking problem to an unconstrained augmented optimal regulation problem, and allows the consideration of general stage cost functions. Then, a novel multi-step VI algorithm based on Q-learning and linear programming is derived. The proposed algorithm improves the convergence speed of VI, avoids the requirement for an initial stabilizing control policy of PI, and computes a constrained optimal feedback controller without the knowledge of a system model and stage cost function. Simulation studies demonstrate the reliability and performance of the proposed approach.