Stance Detection in Web and Social Media: A Comparative Study

TL;DR

This paper systematically compares various stance detection models on social media and online news datasets, assessing their reproducibility and performance to identify strengths and weaknesses in current approaches.

Contribution

It provides the first comprehensive reproducibility analysis and comparative evaluation of neural and classical stance detection models on multiple datasets.

Findings

Neural models generally outperform classical classifiers.

Reproducibility issues are prevalent across existing models.

Performance varies significantly depending on dataset and method.

Abstract

Online forums and social media platforms are increasingly being used to discuss topics of varying polarities where different people take different stances. Several methodologies for automatic stance detection from text have been proposed in literature. To our knowledge, there has not been any systematic investigation towards their reproducibility, and their comparative performances. In this work, we explore the reproducibility of several existing stance detection models, including both neural models and classical classifier-based models. Through experiments on two datasets -- (i)~the popular SemEval microblog dataset, and (ii)~a set of health-related online news articles -- we also perform a detailed comparative analysis of various methods and explore their shortcomings. Implementations of all algorithms discussed in this paper are available at https://github.com/prajwal1210/Stance-Detection-in-Web-and-Social-Media.