Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization

TL;DR

CBOP introduces a conservative Bayesian approach for offline RL that adaptively combines model-based and model-free estimates based on uncertainty, significantly improving performance over prior methods.

Contribution

It proposes a novel conservative Bayesian value expansion method that effectively balances model reliance and uncertainty in offline policy optimization.

Findings

Outperforms previous model-based methods like MOPO, MOReL, and COMBO significantly.

Achieves state-of-the-art results on 11 out of 18 D4RL benchmarks.

Demonstrates robust performance across diverse offline RL datasets.

Abstract

Offline reinforcement learning (RL) addresses the problem of learning a performant policy from a fixed batch of data collected by following some behavior policy. Model-based approaches are particularly appealing in the offline setting since they can extract more learning signals from the logged dataset by learning a model of the environment. However, the performance of existing model-based approaches falls short of model-free counterparts, due to the compounding of estimation errors in the learned model. Driven by this observation, we argue that it is critical for a model-based method to understand when to trust the model and when to rely on model-free estimates, and how to act conservatively w.r.t. both. To this end, we derive an elegant and simple methodology called conservative Bayesian model-based value expansion for offline policy optimization (CBOP), that trades off model-free and model-based estimates during the policy evaluation step according to their epistemic uncertainties, and facilitates conservatism by taking a lower bound on the Bayesian posterior value estimate. On the standard D4RL continuous control tasks, we find that our method significantly outperforms previous model-based approaches: e.g., MOPO by $116.4$%, MOReL by $23.2$% and COMBO by $23.7$%. Further, CBOP achieves state-of-the-art performance on $11$ out of $18$ benchmark datasets while doing on par on the remaining datasets.