Distributional Offline Policy Evaluation with Predictive Error Guarantees

TL;DR

This paper introduces Fitted Likelihood Estimation (FLE), an algorithm for distributional offline policy evaluation that leverages probabilistic generative models and provides theoretical guarantees on distribution closeness.

Contribution

The paper proposes FLE, a flexible MLE-based method for distributional offline policy evaluation with theoretical guarantees and practical validation using advanced generative models.

Findings

FLE achieves close distributional estimates under total variation and Wasserstein distances.

FLE with diffusion models effectively estimates complex multi-dimensional reward distributions.

Theoretical guarantees depend on data coverage and successful MLE training.

Abstract

We study the problem of estimating the distribution of the return of a policy using an offline dataset that is not generated from the policy, i.e., distributional offline policy evaluation (OPE). We propose an algorithm called Fitted Likelihood Estimation (FLE), which conducts a sequence of Maximum Likelihood Estimation (MLE) and has the flexibility of integrating any state-of-the-art probabilistic generative models as long as it can be trained via MLE. FLE can be used for both finite-horizon and infinite-horizon discounted settings where rewards can be multi-dimensional vectors. Our theoretical results show that for both finite-horizon and infinite-horizon discounted settings, FLE can learn distributions that are close to the ground truth under total variation distance and Wasserstein distance, respectively. Our theoretical results hold under the conditions that the offline data covers the test policy's traces and that the supervised learning MLE procedures succeed. Experimentally, we demonstrate the performance of FLE with two generative models, Gaussian mixture models and diffusion models. For the multi-dimensional reward setting, FLE with diffusion models is capable of estimating the complicated distribution of the return of a test policy.