Characterizing the ability of LLMs to recapitulate Americans' distributional responses to public opinion polling questions across political issues

TL;DR

This paper introduces a new LLM-based framework for predicting public opinion distributions on political issues, demonstrating higher accuracy and lower cost than traditional survey methods and previous LLM approaches.

Contribution

The authors develop and evaluate a novel framework that predicts response distributions directly, improving accuracy and predictability over existing LLM polling methods.

Findings

The proposed framework outperforms individual querying in accuracy.

It is significantly more cost-effective than traditional polling.

Performance varies predictably across demographics and questions.

Abstract

Traditional survey-based political issue polling is becoming less tractable due to increasing costs and risk of bias associated with growing non-response rates and declining coverage of key demographic groups. With researchers and pollsters seeking alternatives, Large Language Models have drawn attention for their potential to augment human population studies in polling contexts. We propose and implement a new framework for anticipating human responses on multiple-choice political issue polling questions by directly prompting an LLM to predict a distribution of responses. By comparison to a large and high quality issue poll of the US population, the Cooperative Election Study, we evaluate how the accuracy of this framework varies across a range of demographics and questions on a variety of topics, as well as how this framework compares to previously proposed frameworks where LLMs are repeatedly queried to simulate individual respondents. We find the proposed framework consistently exhibits more accurate predictions than individual querying at significantly lower cost. In addition, we find the performance of the proposed framework varies much more systematically and predictably across demographics and questions, making it possible for those performing AI polling to better anticipate model performance using only information available before a query is issued.