Leveraging Factored Action Spaces for Off-Policy Evaluation

TL;DR

This paper introduces a method to improve off-policy evaluation by decomposing large action spaces into smaller, independent sub-actions, reducing variance without increasing bias, and verified through theoretical analysis and simulations.

Contribution

It proposes a new family of decomposed importance sampling estimators leveraging factored action spaces, enhancing OPE accuracy in large action spaces.

Findings

Decomposed IS estimators have lower variance than traditional estimators.

Theoretical proof of zero bias preservation under certain assumptions.

Empirical validation confirms variance reduction benefits.

Abstract

Off-policy evaluation (OPE) aims to estimate the benefit of following a counterfactual sequence of actions, given data collected from executed sequences. However, existing OPE estimators often exhibit high bias and high variance in problems involving large, combinatorial action spaces. We investigate how to mitigate this issue using factored action spaces i.e. expressing each action as a combination of independent sub-actions from smaller action spaces. This approach facilitates a finer-grained analysis of how actions differ in their effects. In this work, we propose a new family of "decomposed" importance sampling (IS) estimators based on factored action spaces. Given certain assumptions on the underlying problem structure, we prove that the decomposed IS estimators have less variance than their original non-decomposed versions, while preserving the property of zero bias. Through simulations, we empirically verify our theoretical results, probing the validity of various assumptions. Provided with a technique that can derive the action space factorisation for a given problem, our work shows that OPE can be improved "for free" by utilising this inherent problem structure.