Off-Policy Evaluation in Embedded Spaces

TL;DR

This paper introduces a new off-policy evaluation method that uses embedded action spaces and normalizing flows to address positivity violations and feasibility issues in large, non-probabilistic recommendation systems.

Contribution

It proposes the featurized embedded permutation weighting estimator, combining action embeddings and density ratio estimation via normalizing flows for improved off-policy evaluation.

Findings

Reduces positivity violations in large action spaces.

Enables density ratio estimation without explicit policy probabilities.

Demonstrates practical feasibility with recent density estimation techniques.

Abstract

Off-policy evaluation methods are important in recommendation systems and search engines, where data collected under an existing logging policy is used to estimate the performance of a new proposed policy. A common approach to this problem is weighting, where data is weighted by a density ratio between the probability of actions given contexts in the target and logged policies. In practice, two issues often arise. First, many problems have very large action spaces and we may not observe rewards for most actions, and so in finite samples we may encounter a positivity violation. Second, many recommendation systems are not probabilistic and so having access to logging and target policy densities may not be feasible. To address these issues, we introduce the featurized embedded permutation weighting estimator. The estimator computes the density ratio in an action embedding space, which reduces the possibility of positivity violations. The density ratio is computed leveraging recent advances in normalizing flows and density ratio estimation as a classification problem, in order to obtain estimates which are feasible in practice.