Model-based reinforcement learning for infinite-horizon approximate optimal tracking

TL;DR

This paper introduces a model-based reinforcement learning approach for infinite-horizon optimal tracking in nonlinear systems, using concurrent learning to relax excitation conditions and Lyapunov analysis to ensure stability and convergence.

Contribution

It presents a novel online adaptive method combining reinforcement learning with concurrent learning for nonlinear control systems, enabling approximate optimal tracking without persistent excitation.

Findings

Effective tracking of desired trajectories demonstrated in simulations

Convergence to a neighborhood of the optimal policy established

Relaxed excitation conditions compared to traditional methods

Abstract

This paper provides an approximate online adaptive solution to the infinite-horizon optimal tracking problem for control-affine continuous-time nonlinear systems with unknown drift dynamics. Model-based reinforcement learning is used to relax the persistence of excitation condition. Model-based reinforcement learning is implemented using a concurrent learning-based system identifier to simulate experience by evaluating the Bellman error over unexplored areas of the state space. Tracking of the desired trajectory and convergence of the developed policy to a neighborhood of the optimal policy are established via Lyapunov-based stability analysis. Simulation results demonstrate the effectiveness of the developed technique.