Beyond the Return: Off-policy Function Estimation under User-specified Error-measuring Distributions

TL;DR

This paper introduces a flexible regularization approach for off-policy function estimation that guarantees accuracy under user-specified error distributions, relaxing data-coverage assumptions and improving prior methods.

Contribution

It proposes a novel regularizer for marginalized importance sampling that accounts for arbitrary user-defined distributions, enabling accurate off-policy function estimation under weaker assumptions.

Findings

Provides exact characterization of the optimal dual solution.

Regularizer can be modified to relax data-coverage requirements.

Eliminates data-coverage assumptions with strong side information.

Abstract

Off-policy evaluation often refers to two related tasks: estimating the expected return of a policy and estimating its value function (or other functions of interest, such as density ratios). While recent works on marginalized importance sampling (MIS) show that the former can enjoy provable guarantees under realizable function approximation, the latter is only known to be feasible under much stronger assumptions such as prohibitively expressive discriminators. In this work, we provide guarantees for off-policy function estimation under only realizability, by imposing proper regularization on the MIS objectives. Compared to commonly used regularization in MIS, our regularizer is much more flexible and can account for an arbitrary user-specified distribution, under which the learned function will be close to the groundtruth. We provide exact characterization of the optimal dual solution that needs to be realized by the discriminator class, which determines the data-coverage assumption in the case of value-function learning. As another surprising observation, the regularizer can be altered to relax the data-coverage requirement, and completely eliminate it in the ideal case with strong side information.