TruthStance: An Annotated Dataset of Conversations on Truth Social

TL;DR

This paper introduces TruthStance, a comprehensive dataset of Truth Social conversations, along with human annotations and LLM-based labels, to advance argument mining and stance detection on alt-tech platforms.

Contribution

It provides the first large-scale, annotated dataset of Truth Social conversations, including benchmark annotations and LLM-generated labels for argument and stance detection.

Findings

High inter-annotator agreement on annotations

Effective LLM prompting strategies for argument and stance detection

Analysis of argumentation patterns across topics and user depth

Abstract

Argument mining and stance detection are central to understanding how opinions are formed and contested in online discourse. However, most publicly available resources focus on mainstream platforms such as Twitter and Reddit, leaving conversational structure on alt-tech platforms comparatively under-studied. We introduce TruthStance, a large-scale dataset of Truth Social conversation threads spanning 2023-2025, consisting of 24,378 posts and 523,360 comments with reply-tree structure preserved. We provide a human-annotated benchmark of 1,500 instances across argument mining and claim-based stance detection, including inter-annotator agreement, and use it to evaluate large language model (LLM) prompting strategies. Using the best-performing configuration, we release additional LLM-generated labels for 24,352 posts (argument presence) and 107,873 comments (stance to parent), enabling analysis of stance and argumentation patterns across depth, topics, and users. All code and data are released publicly.