Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media

TL;DR

This study evaluates large language models for automated stance detection on complex, low-resource language topics, demonstrating that ChatGPT can perform comparably to supervised models in analyzing media bias over time.

Contribution

It demonstrates the effectiveness of large language models, including ChatGPT, for stance detection in low-resource languages and complex socio-cultural topics, with potential applications in media monitoring.

Findings

Supervised models achieved acceptable stance detection accuracy.

ChatGPT performed similarly to supervised models in zero-shot classification.

The approach successfully analyzed diachronic trends in news corpora.

Abstract

Automated stance detection and related machine learning methods can provide useful insights for media monitoring and academic research. Many of these approaches require annotated training datasets, which limits their applicability for languages where these may not be readily available. This paper explores the applicability of large language models for automated stance detection in a challenging scenario, involving a morphologically complex, lower-resource language, and a socio-culturally complex topic, immigration. If the approach works in this case, it can be expected to perform as well or better in less demanding scenarios. We annotate a large set of pro and anti-immigration examples, and compare the performance of multiple language models as supervised learners. We also probe the usability of ChatGPT as an instructable zero-shot classifier for the same task. Supervised achieves acceptable performance, and ChatGPT yields similar accuracy. This is promising as a potentially simpler and cheaper alternative for text classification tasks, including in lower-resource languages. We further use the best-performing model to investigate diachronic trends over seven years in two corpora of Estonian mainstream and right-wing populist news sources, demonstrating the applicability of the approach for news analytics and media monitoring settings, and discuss correspondences between stance changes and real-world events.