Empirical Bayes Rebiasing

TL;DR

This paper introduces an empirical Bayes rebiasing method to improve the accuracy of interval estimates for many biased and noisy parameters, balancing bias correction and variance.

Contribution

It proposes a novel empirical Bayes approach that learns bias reintroduction from data, providing convergence guarantees and improved inference in practical applications.

Findings

Intervals achieve reliable coverage with estimated bias distribution.

Significant precision gains in LLM evaluation and genetic effect inference.

Method outperforms standard debiasing by reducing interval length.

Abstract

We study methods for simultaneous analysis of many noisy and biased estimates, each paired with an even noisier estimate of its own bias. The analyst's goal is to construct short calibrated intervals for each parameter. The standard debiasing approach, which subtracts the bias estimate from each biased estimate, inflates variance and yields long intervals. In this paper, we propose an empirical Bayes rebiasing strategy that starts from the fully debiased estimates and learns from data how much bias to reintroduce by estimating the unknown bias distribution. We provide convergence rates for the coverage of our intervals when the bias distribution is estimated using nonparametric maximum likelihood. Furthermore, we demonstrate substantial precision gains in prediction-powered inference, including pairwise LLM win-rate evaluations, as well as for inference of direct genetic effects in family-based GWAS.