Online Policies for Real-Time Control Using MRAC-RL

TL;DR

This paper introduces MRAC-RL, a novel framework combining adaptive control and reinforcement learning to develop online control policies that adapt in real-time to modeling errors in dynamical systems, demonstrated on a quadrotor landing task.

Contribution

The paper presents a new MRAC-RL framework with stability guarantees, novel adaptive algorithms, and demonstrates improved real-time control in nonlinear systems over existing RL methods.

Findings

MRAC-RL outperforms state-of-the-art RL algorithms in simulations.

The framework provides stability guarantees for nonlinear control.

Adaptive tracking is successfully achieved in the quadrotor landing task.

Abstract

In this paper, we propose the Model Reference Adaptive Control & Reinforcement Learning (MRAC-RL) approach to developing online policies for systems in which modeling errors occur in real-time. Although reinforcement learning (RL) algorithms have been successfully used to develop control policies for dynamical systems, discrepancies between simulated dynamics and the true target dynamics can cause trained policies to fail to generalize and adapt appropriately when deployed in the real-world. The MRAC-RL framework generates online policies by utilizing an inner-loop adaptive controller together with a simulation-trained outer-loop RL policy. This structure allows MRAC-RL to adapt and operate effectively in a target environment, even when parametric uncertainties exists. We propose a set of novel MRAC algorithms, apply them to a class of nonlinear systems, derive the associated control laws, provide stability guarantees for the resulting closed-loop system, and show that the adaptive tracking objective is achieved. Using a simulation study of an automated quadrotor landing task, we demonstrate that the MRAC-RL approach improves upon state-of-the-art RL algorithms and techniques through the generation of online policies.