Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions

TL;DR

This paper demonstrates that fine-tuning large language models on a large, structured survey dataset significantly improves their ability to predict human survey responses across diverse subpopulations, aiding survey design.

Contribution

It introduces a new fine-tuning approach using the SubPOP dataset, enhancing LLMs' accuracy in predicting survey response distributions and generalizing to unseen data.

Findings

Fine-tuning reduces the LLM-human response gap by up to 46%.

Models generalize well to unseen surveys and subpopulations.

Survey-based fine-tuning improves prediction accuracy across diverse groups.

Abstract

Large language models (LLMs) present novel opportunities in public opinion research by predicting survey responses in advance during the early stages of survey design. Prior methods steer LLMs via descriptions of subpopulations as LLMs' input prompt, yet such prompt engineering approaches have struggled to faithfully predict the distribution of survey responses from human subjects. In this work, we propose directly fine-tuning LLMs to predict response distributions by leveraging unique structural characteristics of survey data. To enable fine-tuning, we curate SubPOP, a significantly scaled dataset of 3,362 questions and 70K subpopulation-response pairs from well-established public opinion surveys. We show that fine-tuning on SubPOP greatly improves the match between LLM predictions and human responses across various subpopulations, reducing the LLM-human gap by up to 46% compared to baselines, and achieves strong generalization to unseen surveys and subpopulations. Our findings highlight the potential of survey-based fine-tuning to improve opinion prediction for diverse, real-world subpopulations and therefore enable more efficient survey designs. Our code is available at https://github.com/JosephJeesungSuh/subpop.