Offline Policy Optimization with Posterior Sampling

TL;DR

This paper introduces PSPO, a Bayesian posterior sampling approach for offline RL that balances generalization and robustness without excessive pessimism.

Contribution

It formulates dynamics modeling as Bayesian inference and integrates posterior sampling with constrained policy optimization for improved offline RL.

Findings

PSPO outperforms state-of-the-art baselines on standard benchmarks.

Theoretical convergence of Q-value estimation under posterior sampling is established.

Decomposition into constrained subproblems guarantees monotonic policy improvement.

Abstract

A fundamental challenge in model-based offline reinforcement learning (RL) lies in the trade-off between generalization and robustness against exploitation errors in out-of-distribution (OOD) regions. While OOD samples may capture valid underlying physical dynamics, they also introduce the risk of model exploitation. Existing methods typically address this risk through excessive pessimistic regularization, which ensures robustness but often sacrifices generalization. To overcome this limitation, we propose Posterior Sampling-based Policy Optimization (PSPO), which formulates dynamics modeling as a Bayesian inference process to derive a posterior that explicitly quantifies model fidelity. Through the integration of posterior sampling and constrained policy optimization, our method leverages dynamics-consistent OOD transitions for generalization while ensuring robustness against model exploitation. Theoretically, we formulate Q-value estimation under posterior sampling as a stochastic approximation problem and establish its convergence. We decompose policy optimization into a sequence of constrained subproblems, demonstrating that solving these subproblems guarantees monotonic improvement until convergence. Experiments on standard benchmarks validate that PSPO achieves superior performance compared to state-of-the-art baselines.