Scaling policy iteration based reinforcement learning for unknown discrete-time linear systems

TL;DR

This paper introduces a novel scaling technique for policy iteration in reinforcement learning, enabling optimal control of unknown discrete-time linear systems without requiring an initial stabilizing policy.

Contribution

It proposes model-based and model-free scaling policy iteration algorithms that do not need an initial stabilizing control, broadening the applicability of RL in control systems.

Findings

Algorithms successfully find optimal control gains from any initial policy.

Numerical results validate the effectiveness of the proposed methods.

Theoretical analysis confirms convergence and stability.

Abstract

In optimal control problem, policy iteration (PI) is a powerful reinforcement learning (RL) tool used for designing optimal controller for the linear systems. However, the need for an initial stabilizing control policy significantly limits its applicability. To address this constraint, this paper proposes a novel scaling technique, which progressively brings a sequence of stable scaled systems closer to the original system, enabling the acquisition of stable control gain. Based on the designed scaling update law, we develop model-based and model-free scaling policy iteration (SPI) algorithms for solving the optimal control problem for discrete-time linear systems, in both known and completely unknown system dynamics scenarios. Unlike existing works on PI based RL, the SPI algorithms do not necessitate an initial stabilizing gain to initialize the algorithms, they can achieve the optimal control under any initial control gain. Finally, the numerical results validate the theoretical findings and confirm the effectiveness of the algorithms.