Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study

TL;DR

This study evaluates how well large language models can generate synthetic German public opinions that reflect real socio-cultural nuances, highlighting differences among models and the impact of prompt variables.

Contribution

It introduces a method to assess LLMs' ability to replicate nuanced public opinions using demographic prompts and compares model performances on German subpopulations.

Findings

Llama outperforms other LLMs in representing subpopulations.

Models better represent left-leaning party supporters than right-leaning.

Prompt variables significantly influence model predictions.

Abstract

In recent research, large language models (LLMs) have been increasingly used to investigate public opinions. This study investigates the algorithmic fidelity of LLMs, i.e., the ability to replicate the socio-cultural context and nuanced opinions of human participants. Using open-ended survey data from the German Longitudinal Election Studies (GLES), we prompt different LLMs to generate synthetic public opinions reflective of German subpopulations by incorporating demographic features into the persona prompts. Our results show that Llama performs better than other LLMs at representing subpopulations, particularly when there is lower opinion diversity within those groups. Our findings further reveal that the LLM performs better for supporters of left-leaning parties like The Greens and The Left compared to other parties, and matches the least with the right-party AfD. Additionally, the inclusion or exclusion of specific variables in the prompts can significantly impact the models' predictions. These findings underscore the importance of aligning LLMs to more effectively model diverse public opinions while minimizing political biases and enhancing robustness in representativeness.