Approximation and Progressive Display of Multiverse Analyses

TL;DR

This paper introduces approximation algorithms and visualization tools for efficiently analyzing multiverse analyses, enabling faster sensitivity estimation and early stopping to improve robustness and transparency.

Contribution

It presents novel sampling-based approximation algorithms and visualizations for monitoring multiverse analyses, reducing computation time and aiding early decision-making.

Findings

Sampling algorithms converge quickly to rank sensitive decisions

Round robin and sketching are twice as fast as uniform sampling

20% of the full multiverse suffices for accurate sensitivity estimates

Abstract

A multiverse analysis evaluates all combinations of "reasonable" analytic decisions to promote robustness and transparency, but can lead to a combinatorial explosion of analyses to compute. Long delays before assessing results prevent users from diagnosing errors and iterating early. We contribute (1) approximation algorithms for estimating multiverse sensitivity and (2) monitoring visualizations for assessing progress and controlling execution on the fly. We evaluate how quickly three sampling-based algorithms converge to accurately rank sensitive decisions in both synthetic and real multiverse analyses. Compared to uniform random sampling, round robin and sketching approaches are 2 times faster in the best case, while on average estimating sensitivity accurately using 20% of the full multiverse. To enable analysts to stop early to fix errors or decide when results are "good enough" to move forward, we visualize both effect size and decision sensitivity estimates with confidence intervals, and surface potential issues including runtime warnings and model quality metrics.