Technical Report: Adaptive Control for Linearizable Systems Using On-Policy Reinforcement Learning

TL;DR

This paper introduces a model-free reinforcement learning framework for adaptive control of unknown linearizable systems, enabling stable tracking without requiring invertible inverse models, and demonstrates its effectiveness through simulation.

Contribution

It presents a novel reinforcement learning approach for adaptive control that relaxes invertibility constraints and provides probabilistic stability guarantees.

Findings

High-probability convergence of tracking errors

Applicability to systems with non-invertible models

Successful simulation on a double pendulum

Abstract

This paper proposes a framework for adaptively learning a feedback linearization-based tracking controller for an unknown system using discrete-time model-free policy-gradient parameter update rules. The primary advantage of the scheme over standard model-reference adaptive control techniques is that it does not require the learned inverse model to be invertible at all instances of time. This enables the use of general function approximators to approximate the linearizing controller for the system without having to worry about singularities. However, the discrete-time and stochastic nature of these algorithms precludes the direct application of standard machinery from the adaptive control literature to provide deterministic stability proofs for the system. Nevertheless, we leverage these techniques alongside tools from the stochastic approximation literature to demonstrate that with high probability the tracking and parameter errors concentrate near zero when a certain persistence of excitation condition is satisfied. A simulated example of a double pendulum demonstrates the utility of the proposed theory. 1