Bayesian Prediction-Powered Inference

TL;DR

This paper introduces a Bayesian framework for prediction-powered inference (PPI) that enhances statistical estimates by combining limited human labels with larger, automatic system-labeled datasets, enabling more accurate and flexible inference methods.

Contribution

The paper presents a Bayesian inference-based framework for PPI, facilitating the development of new, task-specific PPI methods for various types of automatic raters.

Findings

Developed improved PPI methods for discrete autoraters.

Created PPI techniques for non-linear score relationships.

Demonstrated tighter confidence intervals with the proposed methods.

Abstract

Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data. Specifically, PPI methods provide tighter confidence intervals by combining small amounts of human-labeled data with larger amounts of data labeled by a reasonably accurate, but potentially biased, automatic system. We propose a framework for PPI based on Bayesian inference that allows researchers to develop new task-appropriate PPI methods easily. Exploiting the ease with which we can design new metrics, we propose improved PPI methods for several importantcases, such as autoraters that give discrete responses (e.g., prompted LLM ``judges'') and autoraters with scores that have a non-linear relationship to human scores.