Do LLMs Track Public Opinion? A Multi-Model Study of Favorability Predictions in the 2024 U.S. Presidential Election

TL;DR

This study evaluates whether large language models can accurately predict public opinion during the 2024 U.S. presidential election by comparing their favorability predictions to reputable polls, revealing systematic biases and limitations.

Contribution

It provides a comprehensive multi-model analysis of LLMs' ability to track public opinion, highlighting persistent biases and the inadequacy of straightforward querying for election forecasting.

Findings

LLMs systematically overpredict favorability for Kamala Harris.

Biases are smaller and more poll-dependent for Donald Trump.

Predictions are not corrected by temporal smoothing or internet retrieval.

Abstract

We investigate whether Large Language Models (LLMs) can track public opinion as measured by exit polls during the 2024 U.S. presidential election cycle. Our analysis focuses on headline favorability (e.g., "Favorable" vs. "Unfavorable") of presidential candidates across multiple LLMs queried daily throughout the election season. Using the publicly available llm-election-data-2024 dataset, we evaluate predictions from nine LLM configurations against a curated set of five high-quality polls from major organizations including Reuters, CNN, Gallup, Quinnipiac, and ABC. We find systematic directional miscalibration. For Kamala Harris, all models overpredict favorability by 10-40% relative to polls. For Donald Trump, biases are smaller (5-10%) and poll-dependent, with substantially lower cross-model variation. These deviations persist under temporal smoothing and are not corrected by internet-augmented retrieval. We conclude that off-the-shelf LLMs do not reliably track polls when queried in a straightforward manner and discuss implications for election forecasting.