Practical Guidelines for Ideology Detection Pipelines and Psychosocial Applications

TL;DR

This paper offers practical guidelines and a robust pipeline for online ideology detection, addressing data collection challenges, bias quantification, and applying psychosocial theories to large-scale datasets to improve understanding of ideological patterns.

Contribution

It introduces a framework for ideology signal selection, evaluates bias, and demonstrates a high-performance detection pipeline on large datasets, bridging offline psychosocial insights with online applications.

Findings

Right-wing ideologies use more vice-moral language

Greater association of national flags with certain ideologies

Pipeline outperforms state-of-the-art with 0.95 AUC ROC

Abstract

Online ideology detection is crucial for downstream tasks, like countering ideologically motivated violent extremism and modeling opinion dynamics. However, two significant issues arise in practitioners' deployment. Firstly, gold-standard training data is prohibitively labor-intensive to collect and has limited reusability beyond its collection context (i.e., time, location, and platform). Secondly, to circumvent expense, researchers employ ideological signals (such as hashtags shared). Unfortunately, these signals' annotation requirements and context transferability are largely unknown, and the bias they induce remains unquantified. This study provides guidelines for practitioners requiring real-time detection of left, right, and extreme ideologies in large-scale online settings. We propose a framework for pipeline constructions, describing ideology signals by their associated labor and context transferability. We evaluate many constructions, quantifying the bias associated with signals and describing a pipeline that outperforms state-of-the-art methods ($0.95$ AUC ROC). We showcase the capabilities of our pipeline on five datasets containing more than 1.12 million users. We set out to investigate whether the findings in the psychosocial literature, developed for the offline environment, apply to the online setting. We evaluate at scale several psychosocial hypotheses that delineate ideologies concerning morality, grievance, nationalism, and dichotomous thinking. We find that right-wing ideologies use more vice-moral language, have more grievance-filled language, exhibit increased black-and-white thinking patterns, and have a greater association with national flags. This research empowers practitioners with guidelines for ideology detection, and case studies for its application, fostering a safer and better understood digital landscape.