AlignSurvey: A Comprehensive Benchmark for Human Preferences Alignment in Social Surveys

TL;DR

AlignSurvey introduces a comprehensive benchmark and dataset suite for evaluating large language models across the entire social survey process, emphasizing alignment, fairness, and cross-cultural validity.

Contribution

It presents the first full pipeline benchmark for social surveys using LLMs, including datasets, evaluation metrics, and reference models to improve alignment and fairness.

Findings

Effective evaluation metrics for survey response fidelity and fairness.

A large, cross-cultural dataset with 44K+ dialogues and 400K+ survey records.

Open-source models and tools for transparent social survey simulation.

Abstract

Understanding human attitudes, preferences, and behaviors through social surveys is essential for academic research and policymaking. Yet traditional surveys face persistent challenges, including fixed-question formats, high costs, limited adaptability, and difficulties ensuring cross-cultural equivalence. While recent studies explore large language models (LLMs) to simulate survey responses, most are limited to structured questions, overlook the entire survey process, and risks under-representing marginalized groups due to training data biases. We introduce AlignSurvey, the first benchmark that systematically replicates and evaluates the full social survey pipeline using LLMs. It defines four tasks aligned with key survey stages: social role modeling, semi-structured interview modeling, attitude stance modeling and survey response modeling. It also provides task-specific evaluation metrics to assess alignment fidelity, consistency, and fairness at both individual and group levels, with a focus on demographic diversity. To support AlignSurvey, we construct a multi-tiered dataset architecture: (i) the Social Foundation Corpus, a cross-national resource with 44K+ interview dialogues and 400K+ structured survey records; and (ii) a suite of Entire-Pipeline Survey Datasets, including the expert-annotated AlignSurvey-Expert (ASE) and two nationally representative surveys for cross-cultural evaluation. We release the SurveyLM family, obtained through two-stage fine-tuning of open-source LLMs, and offer reference models for evaluating domain-specific alignment. All datasets, models, and tools are available at github and huggingface to support transparent and socially responsible research.