Online Algorithms and Policies Using Adaptive and Machine Learning Approaches

TL;DR

This paper introduces a novel control framework combining Reinforcement Learning and Adaptive Control to achieve real-time stability and parameter learning in nonlinear dynamic systems with uncertainties, validated on a quadrotor landing task.

Contribution

It proposes AC-RL controllers for control-affine systems that ensure stability and enable parameter learning, integrating RL with adaptive control in a novel way.

Findings

Controllers guarantee stability under parametric uncertainties

The approach enables real-time parameter learning with persistent excitation

Validated through numerical experiments on a quadrotor landing task

Abstract

This paper considers the problem of real-time control and learning in dynamic systems subjected to parametric uncertainties. We propose a combination of a Reinforcement Learning (RL) based policy in the outer loop suitably chosen to ensure stability and optimality for the nominal dynamics, together with Adaptive Control (AC) in the inner loop so that in real-time AC contracts the closed-loop dynamics towards a stable trajectory traced out by RL. Two classes of nonlinear dynamic systems are considered, both of which are control-affine. The first class of dynamic systems utilizes equilibrium points %with expansion forms around these points and a Lyapunov approach while second class of nonlinear systems uses contraction theory. AC-RL controllers are proposed for both classes of systems and shown to lead to online policies that guarantee stability using a high-order tuner and accommodate parametric uncertainties and magnitude limits on the input. In addition to establishing a stability guarantee with real-time control, the AC-RL controller is also shown to lead to parameter learning with persistent excitation for the first class of systems. Numerical validations of all algorithms are carried out using a quadrotor landing task on a moving platform.