Iterative Learning Control-Informed Reinforcement Learning for Batch Process Control

TL;DR

This paper proposes a novel framework combining Iterative Learning Control with Reinforcement Learning, using Kalman filters to improve safety, stability, and operational constraint satisfaction in batch process control.

Contribution

It introduces IL-CIRL, integrating ILC with DRL and Kalman filtering to enhance stability and constraint handling in batch process control.

Findings

Improved stability guarantees for DRL in batch processes.

Enhanced disturbance compensation through iterative learning.

Systematic design methodology for safe DRL controllers.

Abstract

A significant limitation of Deep Reinforcement Learning (DRL) is the stochastic uncertainty in actions generated during exploration-exploitation, which poses substantial safety risks during both training and deployment. In industrial process control, the lack of formal stability and convergence guarantees further inhibits adoption of DRL methods by practitioners. Conversely, Iterative Learning Control (ILC) represents a well-established autonomous control methodology for repetitive systems, particularly in batch process optimization. ILC achieves desired control performance through iterative refinement of control laws, either between consecutive batches or within individual batches, to compensate for both repetitive and non-repetitive disturbances. This study introduces an Iterative Learning Control-Informed Reinforcement Learning (IL-CIRL) framework for training DRL controllers in dual-layer batch-to-batch and within-batch control architectures for batch processes. The proposed method incorporates Kalman filter-based state estimation within the iterative learning structure to guide DRL agents toward control policies that satisfy operational constraints and ensure stability guarantees. This approach enables the systematic design of DRL controllers for batch processes operating under multiple disturbance conditions.