High confidence inference on the probability an individual benefits from treatment using experimental or observational data with known propensity scores

TL;DR

This paper develops statistical guarantees for estimating bounds on the probability that an individual benefits from treatment, applicable in both experimental and observational studies with known propensity scores, aiding in understanding heterogeneity and inference precision.

Contribution

It introduces methods for bounding and assessing the uncertainty of individual treatment benefit probabilities using assumption-lean, computationally efficient techniques in various study settings.

Findings

Provides confidence guarantees for bounds on individual benefit probabilities.

Enables heterogeneity analysis across subgroups based on pre-treatment features.

Demonstrates practical application with real randomized experiment data.

Abstract

We seek to understand the probability an individual benefits from treatment (PIBT), an inestimable quantity that must be bounded in practice. Given the innate uncertainty in the population-level bounds on PIBT, we seek to better understand the margin of error for their estimation in order to discern whether the estimated bounds on PIBT are tight or wide due to random chance or not. Toward this goal, we present guarantees to the estimation of bounds on marginal PIBT, with any threshold of interest, for a randomized experiment setting or an observational setting where propensity scores are known. We also derive results that permit us to understand heterogeneity in PIBT across learnable sub-groups delineated by pre-treatment features. These results can be used to help with formal statistical power analyses and frequentist confidence statements for settings where we are interested in assumption-lean inference on PIBT through the target bounds with minimal computational complexity compared to a bootstrap approach. Through a real data example from a large randomized experiment, we also demonstrate how our results for PIBT can allow us to understand the practical implication and goodness of fit of an estimate for the conditional average treatment effect (CATE), a function of an individual's baseline covariates.