A Modified Adaptive Data-Enabled Policy Optimization Control to Resolve State Perturbations

TL;DR

This paper introduces PFDeePO, a modified data-driven control algorithm that reduces state perturbations by avoiding probing noise, ensuring stability and improved convergence without relying on external noise inputs.

Contribution

The paper presents a novel perturbation-free modification to DeePO, enabling stable, noise-free convergence by pausing updates near equilibrium and applying scaled multiplicative noise.

Findings

PFDeePO effectively eliminates state perturbations in simulations.

The modified algorithm maintains system stability and performance.

It reduces reliance on probing noise for persistent excitation.

Abstract

This paper proposes modifications to the data-enabled policy optimization (DeePO) algorithm to mitigate state perturbations. DeePO is an adaptive, data-driven approach designed to iteratively compute a feedback gain equivalent to the certainty-equivalence LQR gain. Like other data-driven approaches based on Willems' fundamental lemma, DeePO requires persistently exciting input signals. However, linear state-feedback gains from LQR designs cannot inherently produce such inputs. To address this, probing noise is conventionally added to the control signal to ensure persistent excitation. However, the added noise may induce undesirable state perturbations. We first identify two key issues that jeopardize the desired performance of DeePO when probing noise is not added: the convergence of states to the equilibrium point, and the convergence of the controller to its optimal value. To address these challenges without relying on probing noise, we propose Perturbation-Free DeePO (PFDeePO) built on two fundamental principles. First, the algorithm pauses the control gain updating in DeePO process when system states are near the equilibrium point. Second, it applies a multiplicative noise, scaled by a mean value of $1$ as a gain for the control signal, when the controller converges. This approach minimizes the impact of noise as the system approaches equilibrium while preserving stability. We demonstrate the effectiveness of PFDeePO through simulations, showcasing its ability to eliminate state perturbations while maintaining system performance and stability.