Is External Information Useful for Stance Detection with LLMs?

TL;DR

This paper systematically evaluates the impact of external Wikipedia and web search information on large language models' stance detection, revealing that such information often degrades performance due to bias alignment, contrasting prior BERT-based findings.

Contribution

It provides the first comprehensive analysis of external info effects on LLMs in stance detection, showing performance degradation and highlighting bias issues.

Findings

External info often reduces LLM stance detection accuracy.

LLMs tend to align predictions with provided information's stance.

Fine-tuning partially mitigates but does not eliminate performance decline.

Abstract

In the stance detection task, a text is classified as either favorable, opposing, or neutral towards a target. Prior work suggests that the use of external information, e.g., excerpts from Wikipedia, improves stance detection performance. However, whether or not such information can benefit large language models (LLMs) remains an unanswered question, despite their wide adoption in many reasoning tasks. In this study, we conduct a systematic evaluation on how Wikipedia and web search external information can affect stance detection across eight LLMs and in three datasets with 12 targets. Surprisingly, we find that such information degrades performance in most cases, with macro F1 scores dropping by up to 27.9\%. We explain this through experiments showing LLMs' tendency to align their predictions with the stance and sentiment of the provided information rather than the ground truth stance of the given text. We also find that performance degradation persists with chain-of-thought prompting, while fine-tuning mitigates but does not fully eliminate it. Our findings, in contrast to previous literature on BERT-based systems which suggests that external information enhances performance, highlight the risks of information biases in LLM-based stance classifiers. Code is available at https://github.com/ngqm/acl2025-stance-detection.