The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions

TL;DR

This paper demonstrates that using LLM-generated synthetic data can significantly enhance stance detection in online political discussions, reducing the need for extensive labeled data and improving model performance.

Contribution

It introduces a method to generate synthetic data with LLMs for stance detection, and shows how selecting informative samples further boosts accuracy with less labeled data.

Findings

Synthetic data improves stance detection accuracy.

Selective sampling enhances performance with less data.

Fine-tuning on synthetic and informative data surpasses baseline.

Abstract

Stance detection holds great potential to improve online political discussions through its deployment in discussion platforms for purposes such as content moderation, topic summarization or to facilitate more balanced discussions. Typically, transformer-based models are employed directly for stance detection, requiring vast amounts of data. However, the wide variety of debate topics in online political discussions makes data collection particularly challenging. LLMs have revived stance detection, but their online deployment in online political discussions faces challenges like inconsistent outputs, biases, and vulnerability to adversarial attacks. We show how LLM-generated synthetic data can improve stance detection for online political discussions by using reliable traditional stance detection models for online deployment, while leveraging the text generation capabilities of LLMs for synthetic data generation in a secure offline environment. To achieve this, (i) we generate synthetic data for specific debate questions by prompting a Mistral-7B model and show that fine-tuning with the generated synthetic data can substantially improve the performance of stance detection, while remaining interpretable and aligned with real world data. (ii) Using the synthetic data as a reference, we can improve performance even further by identifying the most informative samples in an unlabelled dataset, i.e., those samples which the stance detection model is most uncertain about and can benefit from the most. By fine-tuning with both synthetic data and the most informative samples, we surpass the performance of the baseline model that is fine-tuned on all true labels, while labelling considerably less data.