Robust Sampling for Active Statistical Inference

TL;DR

This paper introduces robust sampling strategies for active statistical inference that guarantee performance not worse than uniform sampling and improve accuracy when uncertainty estimates are reliable, demonstrated on social science datasets.

Contribution

It proposes a robust sampling method that interpolates between uniform and active sampling, ensuring stable and improved inference performance.

Findings

Robust sampling guarantees no worse performance than uniform sampling.

The method outperforms standard active inference with reliable uncertainty estimates.

Validated on real datasets from social science and survey research.

Abstract

Active statistical inference is a new method for inference with AI-assisted data collection. Given a budget on the number of labeled data points that can be collected and assuming access to an AI predictive model, the basic idea is to improve estimation accuracy by prioritizing the collection of labels where the model is most uncertain. The drawback, however, is that inaccurate uncertainty estimates can make active sampling produce highly noisy results, potentially worse than those from naive uniform sampling. In this work, we present robust sampling strategies for active statistical inference. Robust sampling ensures that the resulting estimator is never worse than the estimator using uniform sampling. Furthermore, with reliable uncertainty estimates, the estimator usually outperforms standard active inference. This is achieved by optimally interpolating between uniform and active sampling, depending on the quality of the uncertainty scores, and by using ideas from robust optimization. We demonstrate the utility of the method on a series of real datasets from computational social science and survey research.