Uncertainty-Aware Estimation of Mis/Disinformation Prevalence on Social Media

TL;DR

This paper introduces a comprehensive method to quantify various uncertainties in estimating social media mis/disinformation prevalence, emphasizing the importance of uncertainty-aware analysis for robust results.

Contribution

It presents a novel approach combining confidence intervals, simulation, and bootstrapping to jointly quantify multiple sources of uncertainty in mis/disinformation estimation.

Findings

Keyword-based data retrieval increases variability in estimates.

Uncertainty quantification reveals wider confidence intervals.

Joint analysis of uncertainties improves robustness of prevalence estimates.

Abstract

Estimation of mis/disinformation prevalence in social media is crucial for designing mitigation strategies to limit its impact. Yet, such estimations are subject to several uncertainties that are rarely quantified jointly. In this study, we present a methodological contribution in which confidence intervals were used to quantify uncertainties related to mis/disinformation prevalence. The analysis draws on a multi-platform, multilingual dataset annotated by professional fact-checkers. Data were collected between March and April 2025 from Facebook, Instagram, LinkedIn, TikTok, X/Twitter, and YouTube across four EU Member States (France, Poland, Slovakia, and Spain). We account for different causes of uncertainty: (i) sample uncertainty, (ii) annotation uncertainty arising from human disagreement and misclassification, and (iii) data retrieval uncertainty induced by keyword-based data collection. First, we estimate the uncertainty arising from the different causes separately using confidence intervals, simulation-based methods, and bootstrapping. Finally, we combined multinomial simulations of annotator behaviour with keyword and post-resampling to capture the joint impact of measurement uncertainty on mis/disinformation prevalence estimates. The proposed methodological approach highlights the importance of uncertainty-aware estimation of mis/disinformation prevalence for robust analysis. The empirical results of this study show that keyword-based data retrieval can exceed baseline variability, leading to wider confidence intervals around prevalence estimates.