LLMs Can Infer Political Alignment from Online Conversations

TL;DR

This paper demonstrates that large language models can accurately infer users' political alignment from online conversations, revealing significant privacy risks and surpassing traditional methods in prediction accuracy.

Contribution

It shows that LLMs can reliably infer political traits from social data, highlighting their capacity to exploit socio-cultural cues and the associated privacy concerns.

Findings

LLMs outperform traditional models in predicting political alignment.

Aggregating multiple inferences improves prediction accuracy.

LLMs use non-explicit political language to infer political orientation.

Abstract

Due to the correlational structure in our traits such as identities, cultures, and political attitudes, seemingly innocuous preferences like following a band or using a specific slang can reveal private traits. This possibility, especially when combined with massive, public social data and advanced computational methods, poses a fundamental privacy risk. As our data exposure online and the rapid advancement of AI are increasing the risk of misuse, it is critical to understand the capacity of large language models (LLMs) to exploit such potential. Here, using online discussions on DebateOrg and Reddit, we show that LLMs can reliably infer hidden political alignment, significantly outperforming traditional machine learning models. Prediction accuracy further improves as we aggregate multiple text-level inferences into a user-level prediction, and as we use more politics-adjacent domains. We demonstrate that LLMs leverage words that are highly predictive of political alignment while not being explicitly political. Our findings underscore the capacity and risks of LLMs for exploiting socio-cultural correlates.