A Moment-Based Generalization to Post-Prediction Inference

TL;DR

This paper introduces a moment-based extension to post-prediction inference methods, improving bias reduction and calibration in AI/ML-generated data analyses by relaxing previous assumptions and maintaining statistical validity.

Contribution

It proposes a simple, unbiased extension to existing post-prediction inference methods that enhances calibration and reduces bias in downstream analysis.

Findings

Maintains nominal Type I error rates

Reduces bias in estimates

Achieves proper coverage in simulations

Abstract

Artificial intelligence (AI) and machine learning (ML) are increasingly used to generate data for downstream analyses, yet naively treating these predictions as true observations can lead to biased results and incorrect inference. Wang et al. (2020) proposed a method, post-prediction inference, which calibrates inference by modeling the relationship between AI/ML-predicted and observed outcomes in a small, gold-standard sample. Since then, several methods have been developed for inference with predicted data. We revisit Wang et al. in light of these recent developments. We reflect on their assumptions and offer a simple extension of their method which relaxes these assumptions. Our extension (1) yields unbiased point estimates under standard conditions and (2) incorporates a simple scaling factor to preserve calibration variability. In extensive simulations, we show that our method maintains nominal Type I error rates, reduces bias, and achieves proper coverage.