Transfer Learning for a Class of Cascade Dynamical Systems

TL;DR

This paper investigates transfer learning for cascade dynamical systems in reinforcement learning, demonstrating that policies trained on reduced models can be effectively deployed on full systems with stability guarantees, supported by quadrotor experiments.

Contribution

It introduces a transfer learning approach for cascade systems, providing theoretical guarantees based on inner loop stability and validating with numerical experiments.

Findings

Transfer learning reduces simulation time for complex systems.

Guarantees depend on the stability of the inner loop controller.

Numerical experiments confirm theoretical results on quadrotors.

Abstract

This work considers the problem of transfer learning in the context of reinforcement learning. Specifically, we consider training a policy in a reduced order system and deploying it in the full state system. The motivation for this training strategy is that running simulations in the full-state system may take excessive time if the dynamics are complex. While transfer learning alleviates the computational issue, the transfer guarantees depend on the discrepancy between the two systems. In this work, we consider a class of cascade dynamical systems, where the dynamics of a subset of the state-space influence the rest of the states but not vice-versa. The reinforcement learning policy learns in a model that ignores the dynamics of these states and treats them as commanded inputs. In the full-state system, these dynamics are handled using a classic controller (e.g., a PID). These systems have vast applications in the control literature and their structure allows us to provide transfer guarantees that depend on the stability of the inner loop controller. Numerical experiments on a quadrotor support the theoretical findings.