Benchmarking Debiasing Methods for LLM-based Parameter Estimates

TL;DR

This paper compares debiasing methods for LLM annotations in estimating population parameters, highlighting their performance tradeoffs and finite sample behaviors to guide practical application.

Contribution

It provides a finite-sample comparison of DSL and PPI debiasing methods, revealing their strengths, weaknesses, and the bias-variance tradeoff in applied research contexts.

Findings

DSL often outperforms PPI in bias reduction and efficiency.

Both methods achieve low bias with large datasets.

Performance varies across datasets, indicating a bias-variance tradeoff.

Abstract

Large language models (LLMs) offer an inexpensive yet powerful way to annotate text, but are often inconsistent when compared with experts. These errors can bias downstream estimates of population parameters such as regression coefficients and causal effects. To mitigate this bias, researchers have developed debiasing methods such as Design-based Supervised Learning (DSL) and Prediction-Powered Inference (PPI), which promise valid estimation by combining LLM annotations with a limited number of expensive expert annotations. Although these methods produce consistent estimates under theoretical assumptions, it is unknown how they compare in finite samples of sizes encountered in applied research. We make two contributions. First, we study how each methods performance scales with the number of expert annotations, highlighting regimes where LLM bias or limited expert labels significantly affect results. Second, we compare DSL and PPI across a range of tasks, finding that although both achieve low bias with large datasets, DSL often outperforms PPI on bias reduction and empirical efficiency, but its performance is less consistent across datasets. Our findings indicate that there is a bias-variance tradeoff at the level of debiasing methods, calling for more research on developing metrics for quantifying their efficiency in finite samples.