Measuring Social Media Polarization Using Large Language Models and Heuristic Rules

TL;DR

This paper introduces a novel framework combining large language models and heuristics to effectively measure affective polarization in social media discussions, revealing event-dependent polarization patterns.

Contribution

It presents a new scalable, interpretable method that integrates LLMs and rule-based scoring to quantify affective polarization in online discourse, surpassing prior sentiment-based approaches.

Findings

Polarization escalates before major events (anticipation-driven)

Affective polarization spikes after high-impact events (reactive)

The framework is scalable and interpretable for large social media datasets

Abstract

Understanding affective polarization in online discourse is crucial for evaluating the societal impact of social media interactions. This study presents a novel framework that leverages large language models (LLMs) and domain-informed heuristics to systematically analyze and quantify affective polarization in discussions on divisive topics such as climate change and gun control. Unlike most prior approaches that relied on sentiment analysis or predefined classifiers, our method integrates LLMs to extract stance, affective tone, and agreement patterns from large-scale social media discussions. We then apply a rule-based scoring system capable of quantifying affective polarization even in small conversations consisting of single interactions, based on stance alignment, emotional content, and interaction dynamics. Our analysis reveals distinct polarization patterns that are event dependent: (i) anticipation-driven polarization, where extreme polarization escalates before well-publicized events, and (ii) reactive polarization, where intense affective polarization spikes immediately after sudden, high-impact events. By combining AI-driven content annotation with domain-informed scoring, our framework offers a scalable and interpretable approach to measuring affective polarization. The source code is publicly available at: https://github.com/hasanjawad001/llm-social-media-polarization.