Variable Gain Gradient Descent-based Reinforcement Learning for Robust Optimal Tracking Control of Uncertain Nonlinear System with Input-Constraints

TL;DR

This paper introduces a variable gain gradient descent method in reinforcement learning to enhance convergence speed and stability in controlling uncertain nonlinear systems with input constraints.

Contribution

It proposes a novel critic neural network tuning law with variable gain gradient descent that adapts learning rates based on HJB error, improving convergence and stability.

Findings

Faster convergence of critic neural network weights.

Tighter residual set for system trajectories.

Validated robustness through numerical simulations.

Abstract

In recent times, a variety of Reinforcement Learning (RL) algorithms have been proposed for optimal tracking problem of continuous time nonlinear systems with input constraints. Most of these algorithms are based on the notion of uniform ultimate boundedness (UUB) stability, in which normally higher learning rates are avoided in order to restrict oscillations in state error to smaller values. However, this comes at the cost of higher convergence time of critic neural network weights. This paper addresses that problem by proposing a novel tuning law containing a variable gain gradient descent for critic neural network that can adjust the learning rate based on Hamilton-Jacobi-Bellman (HJB) error. By allowing high learning rate the proposed variable gain gradient descent tuning law could improve the convergence time of critic neural network weights. Simultaneously, it also results in tighter residual set, on which trajectories of augmented system converge to, leading to smaller oscillations in state error. A tighter bound for UUB stability of the proposed update mechanism is proved. Numerical studies are then presented to validate the robust Reinforcement Learning control scheme in controlling a continuous time nonlinear system.