High-Confidence Off-Policy (or Counterfactual) Variance Estimation

TL;DR

This paper addresses the challenge of providing high-confidence estimates and bounds for the variance of returns in off-policy data, which is crucial for safe deployment in high-risk decision-making systems.

Contribution

It introduces novel methods for high-confidence off-policy variance estimation, filling a gap in existing off-policy evaluation techniques.

Findings

Proposes a new high-confidence variance estimation method

Provides theoretical guarantees for variance bounds

Demonstrates effectiveness on benchmark datasets

Abstract

Many sequential decision-making systems leverage data collected using prior policies to propose a new policy. For critical applications, it is important that high-confidence guarantees on the new policy's behavior are provided before deployment, to ensure that the policy will behave as desired. Prior works have studied high-confidence off-policy estimation of the expected return, however, high-confidence off-policy estimation of the variance of returns can be equally critical for high-risk applications. In this paper, we tackle the previously open problem of estimating and bounding, with high confidence, the variance of returns from off-policy data