Adaptive Observation-Based Efficient Reinforcement Learning for Uncertain Systems

TL;DR

This paper introduces an adaptive observation-based reinforcement learning method for uncertain systems, utilizing a novel concurrent learning adaptive observer to estimate states and parameters without requiring persistence of excitation.

Contribution

It proposes a new CL-AEO that jointly estimates states and parameters without derivative calculations, enabling efficient RL for uncertain systems.

Findings

Converges to system origin and optimal policy without persistence of excitation.

Demonstrates effectiveness through comparative simulations.

Provides theoretical guarantees for practical convergence.

Abstract

This paper develops an adaptive observation-based efficient reinforcement learning (RL) approach for systems with uncertain drift dynamics. A novel concurrent learning adaptive extended observer (CL-AEO) is first designed to jointly estimate the system state and parameter. This observer has a two-timescale structure and doesn't require any additional numerical techniques to calculate the state derivative information. The idea of concurrent learning (CL) is leveraged to use the recorded data, which leads to a relaxed verifiable excitation condition for the convergence of parameter estimation. Based on the estimated state and parameter provided by the CL-AEO, a simulation of experience based RL scheme is developed to online approximate the optimal control policy. Rigorous theoretical analysis is given to show that the practical convergence of the system state to the origin and the developed policy to the ideal optimal policy can be achieved without the persistence of excitation (PE) condition. Finally, the effectiveness and superiority of the developed methodology are demonstrated via comparative simulations.