Generalized Prediction-Powered Inference, with Application to Binary Classifier Evaluation

TL;DR

This paper generalizes prediction-powered inference (PPI) to any regular asymptotically linear estimator, explores its efficiency limitations, and proposes modifications to handle covariate shift, with applications to classifier evaluation.

Contribution

It extends PPI beyond M-estimators, situates it within semi-parametric theory, and introduces modified estimators for covariate shift scenarios.

Findings

PPI does not reach semi-parametric efficiency outside restrictive cases.

Modified PPI estimators effectively handle covariate distribution shifts.

Numerical studies demonstrate accurate estimation of classifier metrics.

Abstract

In the partially-observed outcome setting, a recent set of proposals known as "prediction-powered inference" (PPI) involve (i) applying a pre-trained machine learning model to predict the response, and then (ii) using these predictions to obtain an estimator of the parameter of interest with asymptotic variance no greater than that which would be obtained using only the labeled observations. While existing PPI proposals consider estimators arising from M-estimation, in this paper we generalize PPI to any regular asymptotically linear estimator. Furthermore, by situating PPI within the context of an existing rich literature on missing data and semi-parametric efficiency theory, we show that while PPI does not achieve the semi-parametric efficiency lower bound outside of very restrictive and unrealistic scenarios, it can be viewed as a computationally-simple alternative to proposals in that literature. We exploit connections to that literature to propose modified PPI estimators that can handle three distinct forms of covariate distribution shift. Finally, we illustrate these developments by constructing PPI estimators of true positive rate, false positive rate, and area under the curve via numerical studies.