QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models

TL;DR

QSTN is an open-source framework that systematically generates and evaluates questionnaire responses from large language models, improving survey reliability and reducing computational costs.

Contribution

It introduces a modular, no-code framework for robust questionnaire inference with LLMs, enabling systematic evaluation of prompt and presentation effects.

Findings

Question structure significantly affects response accuracy.

Presentation methods can reduce computational costs.

Response generation methods impact alignment with human answers.

Abstract

We introduce QSTN, an open-source Python framework for systematically generating responses from questionnaire-style prompts to support in-silico surveys and annotation tasks with large language models (LLMs). QSTN enables robust evaluation of questionnaire presentation, prompt perturbations, and response generation methods. Our extensive evaluation (>40 million survey responses) shows that question structure and response generation methods have a significant impact on the alignment of generated survey responses with human answers. We also find that answers can be obtained for a fraction of the compute cost, by changing the presentation method. In addition, we offer a no-code user interface that allows researchers to set up robust experiments with LLMs \emph{without coding knowledge}. We hope that QSTN will support the reproducibility and reliability of LLM-based research in the future.