Overcoming Model Bias for Robust Offline Deep Reinforcement Learning

TL;DR

This paper introduces MOOSE, a model-based offline reinforcement learning algorithm that enhances robustness and stability by using dynamics models to evaluate policies, outperforming existing model-free methods in industrial and control tasks.

Contribution

The paper proposes MOOSE, a novel model-based offline RL method that reduces bias and improves robustness by leveraging dynamics models instead of value functions.

Findings

MOOSE outperforms state-of-the-art model-free offline RL algorithms in most tested scenarios.

MOOSE maintains low model bias by keeping policies within data support.

The approach demonstrates superior robustness in industrial and continuous control benchmarks.

Abstract

State-of-the-art reinforcement learning algorithms mostly rely on being allowed to directly interact with their environment to collect millions of observations. This makes it hard to transfer their success to industrial control problems, where simulations are often very costly or do not exist, and exploring in the real environment can potentially lead to catastrophic events. Recently developed, model-free, offline RL algorithms, can learn from a single dataset (containing limited exploration) by mitigating extrapolation error in value functions. However, the robustness of the training process is still comparatively low, a problem known from methods using value functions. To improve robustness and stability of the learning process, we use dynamics models to assess policy performance instead of value functions, resulting in MOOSE (MOdel-based Offline policy Search with Ensembles), an algorithm which ensures low model bias by keeping the policy within the support of the data. We compare MOOSE with state-of-the-art model-free, offline RL algorithms { BRAC,} BEAR and BCQ on the Industrial Benchmark and MuJoCo continuous control tasks in terms of robust performance, and find that MOOSE outperforms its model-free counterparts in almost all considered cases, often even by far.