Adaptive Budget Allocation in LLM-Augmented Surveys

TL;DR

This paper introduces an adaptive algorithm for allocating limited human verification resources across survey questions to optimize the reliability of LLM-augmented survey responses, reducing waste and improving efficiency.

Contribution

It presents a novel adaptive allocation method that learns question difficulty and LLM reliability in real time without prior knowledge, with proven guarantees and practical validation.

Findings

Reduces human labeling waste from 10-12% to 2-6% on real survey data.

Achieves the same estimation quality with fewer human samples compared to uniform allocation.

Validated on synthetic and real datasets, with formal performance guarantees.

Abstract

Large language models (LLMs) can generate survey responses at low cost, but their reliability varies substantially across questions and is unknown before data collection. Deploying LLMs in surveys still requires costly human responses for verification and correction. How should a limited human-labeling budget be allocated across questions in real time? We propose an adaptive allocation algorithm that learns which questions are hardest for the LLM while simultaneously collecting human responses. Each human label serves a dual role: it improves the estimate for that question and reveals how well the LLM predicts human responses on it. The algorithm directs more budget to questions where the LLM is least reliable, without requiring any prior knowledge of question-level LLM accuracy. We prove that the allocation gap relative to the best possible allocation vanishes as the budget grows, and validate the approach on both synthetic data and a real survey dataset with 68 questions and over 2000 respondents. On real survey data, the standard practice of allocating human labels uniformly across questions wastes 10--12% of the budget relative to the optimal; our algorithm reduces this waste to 2--6%, and the advantage grows as questions become more heterogeneous in LLM prediction quality. The algorithm achieves the same estimation quality as traditional uniform sampling with fewer human samples, requires no pilot study, and is backed by formal performance guarantees validated on real survey data. More broadly, the framework applies whenever scarce human oversight must be allocated across tasks where LLM reliability is unknown.