Is Stance Detection Topic-Independent and Cross-topic Generalizable? -- A Reproduction Study

TL;DR

This study reproduces and analyzes state-of-the-art cross-topic stance detection, revealing that model performance varies across topics and highlighting the importance of addressing topic-specific vocabulary and context for better generalization.

Contribution

It systematically reproduces prior work and investigates the topic-independence and generalizability of stance detection models across diverse topics.

Findings

Model performance varies significantly across different topics.

Topic-specific vocabulary and socio-cultural context influence stance detection accuracy.

Addressing topic-specific features is crucial for improving cross-topic generalization.

Abstract

Cross-topic stance detection is the task to automatically detect stances (pro, against, or neutral) on unseen topics. We successfully reproduce state-of-the-art cross-topic stance detection work (Reimers et. al., 2019), and systematically analyze its reproducibility. Our attention then turns to the cross-topic aspect of this work, and the specificity of topics in terms of vocabulary and socio-cultural context. We ask: To what extent is stance detection topic-independent and generalizable across topics? We compare the model's performance on various unseen topics, and find topic (e.g. abortion, cloning), class (e.g. pro, con), and their interaction affecting the model's performance. We conclude that investigating performance on different topics, and addressing topic-specific vocabulary and context, is a future avenue for cross-topic stance detection.