Representation Bias in Political Sample Simulations with Large Language Models

TL;DR

This paper investigates biases in simulating political opinions with large language models, revealing disparities based on language, demographics, and political regimes, and highlighting areas for bias mitigation in AI-driven social science research.

Contribution

It introduces a methodology to quantify representation biases in political sample simulations using LLMs across multiple datasets and political contexts.

Findings

Better performance for vote choice than public opinion

More accurate in English-speaking countries

More effective in democratic and bipartisan systems

Abstract

This study seeks to identify and quantify biases in simulating political samples with Large Language Models, specifically focusing on vote choice and public opinion. Using the GPT-3.5-Turbo model, we leverage data from the American National Election Studies, German Longitudinal Election Study, Zuobiao Dataset, and China Family Panel Studies to simulate voting behaviors and public opinions. This methodology enables us to examine three types of representation bias: disparities based on the the country's language, demographic groups, and political regime types. The findings reveal that simulation performance is generally better for vote choice than for public opinions, more accurate in English-speaking countries, more effective in bipartisan systems than in multi-partisan systems, and stronger in democratic settings than in authoritarian regimes. These results contribute to enhancing our understanding and developing strategies to mitigate biases in AI applications within the field of computational social science.