Data-driven Outer-Loop Control Using Deep Reinforcement Learning for Trajectory Tracking

TL;DR

This paper introduces a model-free, deep reinforcement learning framework for outer-loop control that enhances trajectory tracking performance in complex systems without requiring system modeling.

Contribution

It proposes a novel deep RL-based outer-loop control method that learns to generate modified references, applicable to systems with delays, uncertainties, and unmodeled dynamics.

Findings

Effective in flight control with human delays

Successfully manages large-scale mean-field control problems

Generalizes to unseen scenarios

Abstract

Reference tracking systems involve a plant that is stabilized by a local feedback controller and a command center that indicates the reference set-point the plant should follow. Typically, these systems are subject to limitations such as disturbances, systems delays, constraints, uncertainties, underperforming controllers, and unmodeled parameters that do not allow them to achieve the desired performance. In situations where it is not possible to redesign the inner-loop system, it is usual to incorporate an outer-loop control that instructs the system to follow a modified reference path such that the resultant path is close to the ideal one. Typically, strategies to design the outer-loop control need to know a model of the system, which can be an unfeasible task. In this paper, we propose a framework based on deep reinforcement learning that can learn a policy to generate a modified reference that improves the system's performance in a non-invasive and model-free fashion. To illustrate the effectiveness of our approach, we present two challenging cases in engineering: a flight control with a pilot model that includes human reaction delays, and a mean-field control problem for a massive number of space-heating devices. The proposed strategy successfully designs a reference signal that works even in situations that were not seen during the learning process.