Bellman Residual Orthogonalization for Offline Reinforcement Learning

TL;DR

This paper introduces a new offline reinforcement learning method based on Bellman residual orthogonalization, providing theoretical guarantees and practical algorithms for policy evaluation and optimization with function approximation.

Contribution

It proposes a novel principle for approximating Bellman equations along test functions, enabling confidence intervals and policy optimization in offline RL with theoretical analysis.

Findings

Provides confidence intervals for off-policy evaluation.

Derives policy optimization guarantees with oracle inequalities.

Offers polynomial-time algorithms with theoretical guarantees.

Abstract

We propose and analyze a reinforcement learning principle that approximates the Bellman equations by enforcing their validity only along an user-defined space of test functions. Focusing on applications to model-free offline RL with function approximation, we exploit this principle to derive confidence intervals for off-policy evaluation, as well as to optimize over policies within a prescribed policy class. We prove an oracle inequality on our policy optimization procedure in terms of a trade-off between the value and uncertainty of an arbitrary comparator policy. Different choices of test function spaces allow us to tackle different problems within a common framework. We characterize the loss of efficiency in moving from on-policy to off-policy data using our procedures, and establish connections to concentrability coefficients studied in past work. We examine in depth the implementation of our methods with linear function approximation, and provide theoretical guarantees with polynomial-time implementations even when Bellman closure does not hold.