Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lens

TL;DR

RefPlan introduces a doubly Bayesian approach to offline model-based reinforcement learning, enhancing policy robustness and adaptability by dynamically updating environment uncertainty during deployment.

Contribution

It proposes a novel Reflect-then-Plan method that unifies uncertainty modeling with planning through Bayesian posterior estimation, improving offline RL performance.

Findings

Significantly improves conservative offline RL policies.

Maintains robustness under high epistemic uncertainty.

Demonstrates resilience to environment changes.

Abstract

Offline reinforcement learning (RL) is crucial when online exploration is costly or unsafe but often struggles with high epistemic uncertainty due to limited data. Existing methods rely on fixed conservative policies, restricting adaptivity and generalization. To address this, we propose Reflect-then-Plan (RefPlan), a novel doubly Bayesian offline model-based (MB) planning approach. RefPlan unifies uncertainty modeling and MB planning by recasting planning as Bayesian posterior estimation. At deployment, it updates a belief over environment dynamics using real-time observations, incorporating uncertainty into MB planning via marginalization. Empirical results on standard benchmarks show that RefPlan significantly improves the performance of conservative offline RL policies. In particular, RefPlan maintains robust performance under high epistemic uncertainty and limited data, while demonstrating resilience to changing environment dynamics, improving the flexibility, generalizability, and robustness of offline-learned policies.