A Fast Convergence Theory for Offline Decision Making

TL;DR

This paper introduces a unified framework and a new algorithm for offline decision making, providing the first generic fast convergence guarantees in function approximation settings for problems like offline RL and OPE.

Contribution

It proposes the DMOF framework and the EDD algorithm, establishing instance-dependent bounds and demonstrating fast convergence with dataset size, supported by a lower bound analysis.

Findings

EOEC measures problem correlation and decreases at rate 1/N with dataset size.

EDD achieves fast convergence guarantees under partial coverage assumptions.

Lower bounds validate the theoretical soundness of the proposed approach.

Abstract

This paper proposes the first generic fast convergence result in general function approximation for offline decision making problems, which include offline reinforcement learning (RL) and off-policy evaluation (OPE) as special cases. To unify different settings, we introduce a framework called Decision Making with Offline Feedback (DMOF), which captures a wide range of offline decision making problems. Within this framework, we propose a simple yet powerful algorithm called Empirical Decision with Divergence (EDD), whose upper bound can be termed as a coefficient named Empirical Offline Estimation Coefficient (EOEC). We show that EOEC is instance-dependent and actually measures the correlation of the problem. When assuming partial coverage in the dataset, EOEC will reduce in a rate of $1/N$ where $N$ is the size of the dataset, endowing EDD with a fast convergence guarantee. Finally, we complement the above results with a lower bound in the DMOF framework, which further demonstrates the soundness of our theory.