Predicting User Stances from Target-Agnostic Information using Large Language Models

TL;DR

This study explores the potential of large language models to predict user stances from social media posts without target-specific training, revealing variability in performance and highlighting the importance of user-level features.

Contribution

It demonstrates that LLMs can predict user stances from target-agnostic posts, emphasizing the role of surface-level and user-level features, and calls for further research on their effectiveness.

Findings

LLMs show potential in stance prediction from target-agnostic data.

Performance varies based on target type, prediction strategy, and data quantity.

Target-agnostic posts contain relevant surface-level and user-level information.

Abstract

We investigate Large Language Models' (LLMs) ability to predict a user's stance on a target given a collection of his/her target-agnostic social media posts (i.e., user-level stance prediction). While we show early evidence that LLMs are capable of this task, we highlight considerable variability in the performance of the model across (i) the type of stance target, (ii) the prediction strategy and (iii) the number of target-agnostic posts supplied. Post-hoc analyses further hint at the usefulness of target-agnostic posts in providing relevant information to LLMs through the presence of both surface-level (e.g., target-relevant keywords) and user-level features (e.g., encoding users' moral values). Overall, our findings suggest that LLMs might offer a viable method for determining public stances towards new topics based on historical and target-agnostic data. At the same time, we also call for further research to better understand LLMs' strong performance on the stance prediction task and how their effectiveness varies across task contexts.