Proximal Policy Optimization for Tracking Control Exploiting Future Reference Information

TL;DR

This paper enhances proximal policy optimization (PPO) for tracking control by integrating future reference information, improving generalization to arbitrary references in reinforcement learning-based control systems.

Contribution

It introduces two variants of PPO that incorporate future reference data, including a novel residual space approach for model-free RL, advancing control performance.

Findings

Outperforms PI controller in tracking tasks.

Improves generalization to arbitrary reference signals.

Introduces a novel residual space for RL with future references.

Abstract

In recent years, reinforcement learning (RL) has gained increasing attention in control engineering. Especially, policy gradient methods are widely used. In this work, we improve the tracking performance of proximal policy optimization (PPO) for arbitrary reference signals by incorporating information about future reference values. Two variants of extending the argument of the actor and the critic taking future reference values into account are presented. In the first variant, global future reference values are added to the argument. For the second variant, a novel kind of residual space with future reference values applicable to model-free reinforcement learning is introduced. Our approach is evaluated against a PI controller on a simple drive train model. We expect our method to generalize to arbitrary references better than previous approaches, pointing towards the applicability of RL to control real systems.