Batch-Adaptive Causal Annotations

TL;DR

This paper introduces an optimized batch sampling method for causal effect estimation with missing or costly outcome data, significantly reducing labeling costs while maintaining accuracy.

Contribution

It derives a closed-form solution for optimal sampling to minimize variance in causal inference with missing outcomes, extending to unstructured data annotations.

Findings

Achieves lower mean-squared error compared to baselines.

Recovers AIPW estimate with fewer labels, saving 75% of labeling budget.

Matches confidence intervals with only 25% of the samples.

Abstract

Estimating the causal effects of interventions is crucial to policy and decision-making, yet outcome data are often missing or subject to non-standard measurement error. While ground-truth outcomes can sometimes be obtained through costly data annotation or follow-up, budget constraints typically allow only a fraction of the dataset to be labeled. We address this challenge by optimizing which data points should be sampled for outcome information in order to improve efficiency in average treatment effect estimation with missing outcomes. We derive a closed-form solution for the optimal batch sampling probability by minimizing the asymptotic variance of a doubly robust estimator for causal inference with missing outcomes. Motivated by our street outreach partners, we extend the framework to costly annotations of unstructured data, such as text or images in healthcare and social services. Across simulated and real-world datasets, including one of outreach interventions in homelessness services, our approach achieves substantially lower mean-squared error and recovers the AIPW estimate with fewer labels than existing baselines. In practice, we show that our method can match confidence intervals obtained with 361 random samples using only 90 optimized samples - saving 75% of the labeling budget.