Can Large Language Models Address Open-Target Stance Detection?

TL;DR

This paper explores the challenge of open-target stance detection using large language models, demonstrating their strengths in target and stance prediction when targets are explicit, but highlighting difficulties with non-explicit targets.

Contribution

It introduces the OTSD task, a realistic setting for stance detection without predefined targets, and evaluates LLMs' performance compared to existing methods, including a new target quality metric.

Findings

LLMs outperform TSE in target generation and stance detection when targets are explicit.

LLMs struggle with non-explicit targets in both generation and detection.

A new metric correlates well with human judgment of target quality.

Abstract

Stance detection (SD) identifies the text position towards a target, typically labeled as favor, against, or none. We introduce Open-Target Stance Detection (OTSD), the most realistic task where targets are neither seen during training nor provided as input. We evaluate Large Language Models (LLMs) from GPT, Gemini, Llama, and Mistral families, comparing their performance to the only existing work, Target-Stance Extraction (TSE), which benefits from predefined targets. Unlike TSE, OTSD removes the dependency of a predefined list, making target generation and evaluation more challenging. We also provide a metric for evaluating target quality that correlates well with human judgment. Our experiments reveal that LLMs outperform TSE in target generation, both when the real target is explicitly and not explicitly mentioned in the text. Similarly, LLMs overall surpass TSE in stance detection for both explicit and non-explicit cases. However, LLMs struggle in both target generation and stance detection when the target is not explicit.