Is Pessimism Provably Efficient for Offline RL?

TL;DR

This paper introduces a pessimistic value iteration algorithm for offline RL that is provably efficient and minimax optimal, even without dataset coverage assumptions, by effectively handling dataset coverage issues through a simple penalty mechanism.

Contribution

It proposes a practical, implementable pessimistic algorithm for offline RL with theoretical guarantees that match lower bounds in linear MDPs, highlighting the importance of pessimism in offline RL.

Findings

PEVI achieves a data-dependent upper bound on suboptimality.

In linear MDPs, PEVI matches the information-theoretic lower bound.

Pessimism helps eliminate spurious correlations from irrelevant trajectories.

Abstract

We study offline reinforcement learning (RL), which aims to learn an optimal policy based on a dataset collected a priori. Due to the lack of further interactions with the environment, offline RL suffers from the insufficient coverage of the dataset, which eludes most existing theoretical analysis. In this paper, we propose a pessimistic variant of the value iteration algorithm (PEVI), which incorporates an uncertainty quantifier as the penalty function. Such a penalty function simply flips the sign of the bonus function for promoting exploration in online RL, which makes it easily implementable and compatible with general function approximators. Without assuming the sufficient coverage of the dataset, we establish a data-dependent upper bound on the suboptimality of PEVI for general Markov decision processes (MDPs). When specialized to linear MDPs, it matches the information-theoretic lower bound up to multiplicative factors of the dimension and horizon. In other words, pessimism is not only provably efficient but also minimax optimal. In particular, given the dataset, the learned policy serves as the "best effort" among all policies, as no other policies can do better. Our theoretical analysis identifies the critical role of pessimism in eliminating a notion of spurious correlation, which emerges from the "irrelevant" trajectories that are less covered by the dataset and not informative for the optimal policy.