Exploiting contextual information to improve stance detection in informal political discourse with LLMs

TL;DR

This paper demonstrates that providing user profile context significantly improves the accuracy of large language models in detecting political stance in informal online discourse, especially when language is sarcastic or ambiguous.

Contribution

The study introduces a method of incorporating user profile summaries into LLM prompts, showing substantial accuracy gains in stance detection on real-world political forum data.

Findings

Contextual prompts improve accuracy by up to 38.5%.

Strategic profile content selection outperforms larger, random contexts.

Up to 74% classification accuracy achieved.

Abstract

This study investigates the use of Large Language Models (LLMs) for political stance detection in informal online discourse, where language is often sarcastic, ambiguous, and context-dependent. We explore whether providing contextual information, specifically user profile summaries derived from historical posts, can improve classification accuracy. Using a real-world political forum dataset, we generate structured profiles that summarize users' ideological leaning, recurring topics, and linguistic patterns. We evaluate seven state-of-the-art LLMs across baseline and context-enriched setups through a comprehensive cross-model evaluation. Our findings show that contextual prompts significantly boost accuracy, with improvements ranging from +17.5\% to +38.5\%, achieving up to 74\% accuracy that surpasses previous approaches. We also analyze how profile size and post selection strategies affect performance, showing that strategically chosen political content yields better results than larger, randomly selected contexts. These findings underscore the value of incorporating user-level context to enhance LLM performance in nuanced political classification tasks.