Evaluating Variance Estimates with Relative Efficiency

TL;DR

This paper introduces a method to evaluate the accuracy of variance estimates in experimentation platforms, proposing a more efficient $t^2$-statistic for detecting issues with confidence intervals.

Contribution

It presents a novel empirical approach to assess variance estimate effectiveness and introduces a $t^2$-statistic that improves detection efficiency over traditional FPR-based methods.

Findings

The $t^2$-statistic outperforms empirical FPR in detecting variance issues.

A/A testing can be enhanced by using more informative statistics.

The proposed method improves reliability diagnostics for experimentation platforms.

Abstract

Experimentation platforms in industry must often deal with customer trust issues. Platforms must prove the validity of their claims as well as catch issues that arise. As a central quantity estimated by experimentation platforms, the validity of confidence intervals is of particular concern. To ensure confidence intervals are reliable, we must understand and diagnose when our variance estimates are biased or noisy, or when the confidence intervals may be incorrect. A common method for this is A/A testing, in which both the control and test arms receive the same treatment. One can then test if the empirical false positive rate (FPR) deviates substantially from the target FPR over many tests. However, this approach turns each A/A test into a simple binary random variable. It is an inefficient estimate of the FPR as it throws away information about the magnitude of each experiment result. We show how to empirically evaluate the effectiveness of statistics that monitor the variance estimates that partly dictate a platform's statistical reliability. We also show that statistics other than empirical FPR are more effective at detecting issues. In particular, we propose a $t^2$-statistic that is more sample efficient.