Finite-State Extreme Effect Variable

TL;DR

This paper introduces a theoretical framework for a finite-state extreme effect variable that captures the maximal impact of a variant on an observable, with applications in online A/B testing.

Contribution

It generalizes the concept of the extreme effect variable to finite states and provides a method for its identification through distribution decomposition.

Findings

Effective in online web search evaluation

Theoretical analysis of effect variable properties

Demonstrated utility in A/B testing scenarios

Abstract

We generalize to the finite-state case the notion of the extreme effect variable $Y$ that accumulates all the effect of a variant variable $V$ observed in changes of another variable $X$. We conduct theoretical analysis and turn the problem of finding of an effect variable into a problem of a simultaneous decomposition of a set of distributions. The states of the extreme effect variable, on the one hand, are minimally affected by the variant variable $V$ and, on the other hand, are extremely different with respect to the observable variable $X$. We apply our technique to online evaluation of a web search engine through A/B testing and show its utility.