Quantile Off-Policy Evaluation via Deep Conditional Generative Learning

TL;DR

This paper introduces a novel doubly-robust method for quantile off-policy evaluation in sequential decision making, leveraging deep generative models to better handle skewed reward distributions and provide more robust policy evaluation.

Contribution

It develops a new quantile-focused OPE estimator using deep conditional generative learning, addressing variability and heavy tails in reward distributions.

Findings

Outperforms classical mean-based OPE estimators in heavy-tailed settings

Demonstrates effectiveness on real-world short-video platform data

Provides asymptotic guarantees for the proposed estimator

Abstract

Off-Policy evaluation (OPE) is concerned with evaluating a new target policy using offline data generated by a potentially different behavior policy. It is critical in a number of sequential decision making problems ranging from healthcare to technology industries. Most of the work in existing literature is focused on evaluating the mean outcome of a given policy, and ignores the variability of the outcome. However, in a variety of applications, criteria other than the mean may be more sensible. For example, when the reward distribution is skewed and asymmetric, quantile-based metrics are often preferred for their robustness. In this paper, we propose a doubly-robust inference procedure for quantile OPE in sequential decision making and study its asymptotic properties. In particular, we propose utilizing state-of-the-art deep conditional generative learning methods to handle parameter-dependent nuisance function estimation. We demonstrate the advantages of this proposed estimator through both simulations and a real-world dataset from a short-video platform. In particular, we find that our proposed estimator outperforms classical OPE estimators for the mean in settings with heavy-tailed reward distributions.