Unit Selection: Case Study and Comparison with A/B Test Heuristic

TL;DR

This paper analyzes Li and Pearl's unit selection model, demonstrating its advantages over A/B test heuristics, and provides simulated use cases to aid correct application and highlight limitations of A/B heuristics.

Contribution

It clarifies the relationship between the unit selection model and A/B test heuristics, showing that heuristics are special cases and offering guidance for proper use.

Findings

A/B test heuristics are special cases of the benefit function.

Li-Pearl's model can identify individuals with desired counterfactual behaviors.

Simulated use cases illustrate correct application and limitations.

Abstract

The unit selection problem defined by Li and Pearl identifies individuals who have desired counterfactual behavior patterns, for example, individuals who would respond positively if encouraged and would not otherwise. Li and Pearl showed by example that their unit selection model is beyond the A/B test heuristics. In this paper, we reveal the essence of the A/B test heuristics, which are exceptional cases of the benefit function defined by Li and Pearl. Furthermore, We provided more simulated use cases of Li-Pearl's unit selection model to help decision-makers apply their model correctly, explaining that A/B test heuristics are generally problematic.