Knowing the Past to Predict the Future: Reinforcement Virtual Learning

TL;DR

This paper introduces a cost-efficient reinforcement learning framework that uses virtual models trained on historical data to predict future states and optimize long-term decisions in uncertain environments like batch process control.

Contribution

It proposes a novel virtual learning framework enabling RL models to evolve using only historical data, reducing the need for expensive environment interactions.

Findings

Outperforms existing state-of-the-art methods in Fed-Batch Process control

Effectively balances long-sight and short-sight rewards

Converges to optimal policies through virtual environment interaction

Abstract

Reinforcement Learning (RL)-based control system has received considerable attention in recent decades. However, in many real-world problems, such as Batch Process Control, the environment is uncertain, which requires expensive interaction to acquire the state and reward values. In this paper, we present a cost-efficient framework, such that the RL model can evolve for itself in a Virtual Space using the predictive models with only historical data. The proposed framework enables a step-by-step RL model to predict the future state and select optimal actions for long-sight decisions. The main focuses are summarized as: 1) how to balance the long-sight and short-sight rewards with an optimal strategy; 2) how to make the virtual model interacting with real environment to converge to a final learning policy. Under the experimental settings of Fed-Batch Process, our method consistently outperforms the existing state-of-the-art methods.