Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations

TL;DR

This paper introduces a fine-tuning approach for large language models to accurately simulate survey response distributions at the country level, aiming to reduce the need for costly survey data collection.

Contribution

It presents the first specialization method for LLMs to simulate survey responses, outperforming other approaches and zero-shot classifiers on diverse, unseen survey data.

Findings

Fine-tuning improves response distribution accuracy

Method outperforms zero-shot classifiers

Models still struggle with unseen questions

Abstract

Large-scale surveys are essential tools for informing social science research and policy, but running surveys is costly and time-intensive. If we could accurately simulate group-level survey results, this would therefore be very valuable to social science research. Prior work has explored the use of large language models (LLMs) for simulating human behaviors, mostly through prompting. In this paper, we are the first to specialize LLMs for the task of simulating survey response distributions. As a testbed, we use country-level results from two global cultural surveys. We devise a fine-tuning method based on first-token probabilities to minimize divergence between predicted and actual response distributions for a given question. Then, we show that this method substantially outperforms other methods and zero-shot classifiers, even on unseen questions, countries, and a completely unseen survey. While even our best models struggle with the task, especially on unseen questions, our results demonstrate the benefits of specialization for simulation, which may accelerate progress towards sufficiently accurate simulation in the future.