Inference with generalizable classifier predictions

TL;DR

This paper develops bootstrap-based methods for valid statistical inference using classifier predictions that generalize across conditions, enabling reliable analysis when human labels are replaced by automated classifiers.

Contribution

It formalizes the inference problem with classifier predictions and introduces bootstrap procedures to ensure valid results with generalizable classifiers.

Findings

Bootstrap methods perform well in simulations.

Methods enable sound inference in live cell imaging case study.

Proposes formal framework for inference with classifier predictions.

Abstract

This paper addresses the problem of making statistical inference about a population that can only be identified through classifier predictions. The problem is motivated by scientific studies in which human labels of a population are replaced by a classifier. For downstream analysis of the population based on classifier predictions to be sound, the predictions must generalize equally across experimental conditions. In this paper, we formalize the task of statistical inference using classifier predictions, and propose bootstrap procedures to allow inference with a generalizable classifier. We demonstrate the performance of our methods through extensive simulations and a case study with live cell imaging data.