Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Search

TL;DR

This paper introduces an iterative batch reinforcement learning method that uses ensemble-based model-based policy search with safety and diversity criteria to improve policies over time in high-risk, cost-sensitive environments.

Contribution

It proposes a novel iterative approach combining safety and diversity in ensemble-based model-based policy search for offline reinforcement learning.

Findings

Effective policy improvement through iterative data collection and learning.

Enhanced safety and diversity lead to more robust policies.

Applicable to industrial control and high-risk applications.

Abstract

Batch reinforcement learning enables policy learning without direct interaction with the environment during training, relying exclusively on previously collected sets of interactions. This approach is, therefore, well-suited for high-risk and cost-intensive applications, such as industrial control. Learned policies are commonly restricted to act in a similar fashion as observed in the batch. In a real-world scenario, learned policies are deployed in the industrial system, inevitably leading to the collection of new data that can subsequently be added to the existing recording. The process of learning and deployment can thus take place multiple times throughout the lifespan of a system. In this work, we propose to exploit this iterative nature of applying offline reinforcement learning to guide learned policies towards efficient and informative data collection during deployment, leading to continuous improvement of learned policies while remaining within the support of collected data. We present an algorithmic methodology for iterative batch reinforcement learning based on ensemble-based model-based policy search, augmented with safety and, importantly, a diversity criterion.