Meta-Reinforcement Learning for Adaptive Control of Second Order Systems

TL;DR

This paper presents a meta-reinforcement learning approach for adaptive control of second order systems, leveraging offline model information to enable automatic adaptation and improved performance in process control.

Contribution

It extends previous meta-RL control strategies to second order systems and PID controllers, integrating offline training with online adaptation for process control.

Findings

Meta-RL can adapt to changes in second order system dynamics.

The approach maintains high control performance across varying system parameters.

Extension from first order to second order systems demonstrated effective adaptability.

Abstract

Meta-learning is a branch of machine learning which aims to synthesize data from a distribution of related tasks to efficiently solve new ones. In process control, many systems have similar and well-understood dynamics, which suggests it is feasible to create a generalizable controller through meta-learning. In this work, we formulate a meta reinforcement learning (meta-RL) control strategy that takes advantage of known, offline information for training, such as a model structure. The meta-RL agent is trained over a distribution of model parameters, rather than a single model, enabling the agent to automatically adapt to changes in the process dynamics while maintaining performance. A key design element is the ability to leverage model-based information offline during training, while maintaining a model-free policy structure for interacting with new environments. Our previous work has demonstrated how this approach can be applied to the industrially-relevant problem of tuning proportional-integral controllers to control first order processes. In this work, we briefly reintroduce our methodology and demonstrate how it can be extended to proportional-integral-derivative controllers and second order systems.