Examining Temporalities on Stance Detection towards COVID-19 Vaccination

TL;DR

This paper investigates how temporal shifts affect stance detection models for COVID-19 vaccination on Twitter, revealing that models perform worse when accounting for time-based data splits, highlighting the need to incorporate temporal factors.

Contribution

It introduces an analysis of temporal concept drift in stance detection models and evaluates transformer-based models with chronological data splits, emphasizing the importance of temporal considerations.

Findings

Significant performance drops with chronological splits

Temporal shifts impact stance detection accuracy

Models need to incorporate temporal dynamics

Abstract

Previous studies have highlighted the importance of vaccination as an effective strategy to control the transmission of the COVID-19 virus. It is crucial for policymakers to have a comprehensive understanding of the public's stance towards vaccination on a large scale. However, attitudes towards COVID-19 vaccination, such as pro-vaccine or vaccine hesitancy, have evolved over time on social media. Thus, it is necessary to account for possible temporal shifts when analysing these stances. This study aims to examine the impact of temporal concept drift on stance detection towards COVID-19 vaccination on Twitter. To this end, we evaluate a range of transformer-based models using chronological (splitting the training, validation, and test sets in order of time) and random splits (randomly splitting these three sets) of social media data. Our findings reveal significant discrepancies in model performance between random and chronological splits in several existing COVID-19-related datasets; specifically, chronological splits significantly reduce the accuracy of stance classification. Therefore, real-world stance detection approaches need to be further refined to incorporate temporal factors as a key consideration.