Adaptive Optimal Control for Reference Tracking Independent of Exo-System Dynamics

TL;DR

This paper introduces an adaptive optimal control method that enables reinforcement learning-based tracking of arbitrary reference trajectories, overcoming limitations of existing methods tied to fixed exo-system dynamics.

Contribution

The paper presents a novel Q-function structure that incorporates reference trajectories directly, allowing for adaptive optimal control independent of exo-system dynamics.

Findings

Effective tracking of non-exo-system generated trajectories demonstrated

Convergent reinforcement learning algorithm developed for optimal control

Real-world examples validate the method's applicability

Abstract

Model-free control based on the idea of Reinforcement Learning is a promising approach that has recently gained extensive attention. However, Reinforcement-Learning-based control methods solely focus on the regulation problem or learn to track a reference that is generated by a time-invariant exo-system. In the latter case, controllers are only able to track the time-invariant reference dynamics which they have been trained on and need to be re-trained each time the reference dynamics change. Consequently, these methods fail in a number of applications which obviously rely on a trajectory not being generated by an exo-system. One prominent example is autonomous driving. This paper provides for the first time an adaptive optimal control method capable to track reference trajectories not being generated by a time-invariant exo-system. The main innovation is a novel Q-function that directly incorporates a given reference trajectory on a moving horizon. This new Q-function exhibits a particular structure which allows the design of an efficient, iterative, provably convergent Reinforcement Learning algorithm that enables optimal tracking. Two real-world examples demonstrate the effectiveness of our new method.