Robust Regularized Policy Iteration under Transition Uncertainty

TL;DR

This paper introduces RRPI, a robust policy iteration method for offline RL that explicitly accounts for transition uncertainty, improving performance and safety under distribution shifts.

Contribution

The paper proposes a novel robust regularized policy iteration framework that handles transition uncertainty with theoretical guarantees and practical efficiency.

Findings

RRPI outperforms recent baselines on D4RL benchmarks.

RRPI maintains robust performance by aligning low Q-values with high uncertainty.

The method guarantees monotonic improvement and convergence in robust policy optimization.

Abstract

Offline reinforcement learning (RL) enables data-efficient and safe policy learning without online exploration, but its performance often degrades under distribution shift. The learned policy may visit out-of-distribution state-action pairs where value estimates and learned dynamics are unreliable. To address policy-induced extrapolation and transition uncertainty in a unified framework, we formulate offline RL as robust policy optimization, treating the transition kernel as a decision variable within an uncertainty set and optimizing the policy against the worst-case dynamics. We propose Robust Regularized Policy Iteration (RRPI), which replaces the intractable max-min bilevel objective with a tractable KL-regularized surrogate and derives an efficient policy iteration procedure based on a robust regularized Bellman operator. We provide theoretical guarantees by showing that the proposed operator is a $\gamma$-contraction and that iteratively updating the surrogate yields monotonic improvement of the original robust objective with convergence. Experiments on D4RL benchmarks demonstrate that RRPI achieves strong average performance, outperforming recent baselines including percentile-based methods on the majority of environments while remaining competitive on the rest. Moreover, RRPI exhibits robust performance by aligning lower $Q$-values with high epistemic uncertainty, which prevents the policy from executing unreliable out-of-distribution actions.