Boba: Authoring and Visualizing Multiverse Analyses

TL;DR

Boba is a combined language and visualization tool that simplifies authoring and reviewing multiverse analyses by managing multiple analysis paths and providing linked visualizations for robustness assessment.

Contribution

It introduces a domain-specific language and visual system for efficient multiverse analysis authoring and review, enhancing robustness and transparency in data analysis.

Findings

Boba enables writing shared analysis code with local variations.

The visualizer links model results with decision space for systematic assessment.

Demonstrated utility through two real-world case studies.

Abstract

Multiverse analysis is an approach to data analysis in which all "reasonable" analytic decisions are evaluated in parallel and interpreted collectively, in order to foster robustness and transparency. However, specifying a multiverse is demanding because analysts must manage myriad variants from a cross-product of analytic decisions, and the results require nuanced interpretation. We contribute Boba: an integrated domain-specific language (DSL) and visual analysis system for authoring and reviewing multiverse analyses. With the Boba DSL, analysts write the shared portion of analysis code only once, alongside local variations defining alternative decisions, from which the compiler generates a multiplex of scripts representing all possible analysis paths. The Boba Visualizer provides linked views of model results and the multiverse decision space to enable rapid, systematic assessment of consequential decisions and robustness, including sampling uncertainty and model fit. We demonstrate Boba's utility through two data analysis case studies, and reflect on challenges and design opportunities for multiverse analysis software.