Making Uncertainty Visible: Multiverse Analysis for Robust Computational Social Science

TL;DR

This paper demonstrates how multiverse analysis enhances robustness and transparency in computational social science by examining methodological variability and failures across three case studies.

Contribution

It introduces the application of multiverse analysis to computational social science, highlighting methodological variability and failure modes in three distinct case studies.

Findings

Multiverse analysis reveals how results vary with different methodological choices.

It uncovers often-ignored computational failures in social science methods.

Provides guidelines for conducting and communicating multiverse analyses.

Abstract

Through case studies, we demonstrate how multiverse analysis can strengthen the robustness and transparency of computational social science findings against alternative methodological decisions. We conduct multiverse analyses of three published social science studies that use the following computational methods: Bayesian analysis, network generative modeling, and machine learning with or without large language models. These methods are applied frequently in computational social science studies, yet entail a greater degree of arbitrariness in terms of methodological choices, or "researcher degrees of freedom." Our multiverse analyses reveal how the empirical findings in these studies vary as a function of various plausible decision combinations. Our three case studies also expose an often-ignored motivation for conducting multiverse analysis: Showing which methodological combinations lead to computational failure. These failed cases are usually not communicated in the published reports, even though these sophisticated computational methods have a much higher likelihood of failure. We end our paper with suggestions on how to find defensible decision combinations for multiverse analysis of computational social science studies and how to communicate multiverse analysis findings fairly.