MFRL-BI: Design of a Model-free Reinforcement Learning Process Control Scheme by Using Bayesian Inference

TL;DR

This paper introduces a model-free reinforcement learning control scheme using Bayesian inference to adaptively optimize manufacturing processes in real-time, especially when process models are inaccurate or unknown.

Contribution

It proposes a novel MFRL control approach that updates disturbance distributions via Bayesian inference, improving process control without relying on predefined models.

Findings

Effective in nonlinear CMP process control

Performs well with unknown process models

Demonstrates theoretical guarantees and numerical efficiency

Abstract

Design of process control scheme is critical for quality assurance to reduce variations in manufacturing systems. Taking semiconductor manufacturing as an example, extensive literature focuses on control optimization based on certain process models (usually linear models), which are obtained by experiments before a manufacturing process starts. However, in real applications, pre-defined models may not be accurate, especially for a complex manufacturing system. To tackle model inaccuracy, we propose a model-free reinforcement learning (MFRL) approach to conduct experiments and optimize control simultaneously according to real-time data. Specifically, we design a novel MFRL control scheme by updating the distribution of disturbances using Bayesian inference to reduce their large variations during manufacturing processes. As a result, the proposed MFRL controller is demonstrated to perform well in a nonlinear chemical mechanical planarization (CMP) process when the process model is unknown. Theoretical properties are also guaranteed when disturbances are additive. The numerical studies also demonstrate the effectiveness and efficiency of our methodology.