MetaExplorer: Facilitating Reasoning with Epistemic Uncertainty in Meta-analysis

TL;DR

MetaExplorer is a new system designed to improve meta-analyses by guiding users through assessing and incorporating epistemic uncertainty, thereby enhancing the validity of evidence synthesis.

Contribution

It introduces a structured, expert-informed workflow for addressing epistemic uncertainty in meta-analysis, which is not well-supported by existing tools.

Findings

Structured workflow increases awareness of epistemic concerns.

Engages users in dialogue about evidence quality and standards.

Potential to improve meta-analytic validity.

Abstract

Scientists often use meta-analysis to characterize the impact of an intervention on some outcome of interest across a body of literature. However, threats to the utility and validity of meta-analytic estimates arise when scientists average over potentially important variations in context like different research designs. Uncertainty about quality and commensurability of evidence casts doubt on results from meta-analysis, yet existing software tools for meta-analysis do not necessarily emphasize addressing these concerns in their workflows. We present MetaExplorer, a prototype system for meta-analysis that we developed using iterative design with meta-analysis experts to provide a guided process for eliciting assessments of uncertainty and reasoning about how to incorporate them during statistical inference. Our qualitative evaluation of MetaExplorer with experienced meta-analysts shows that imposing a structured workflow both elevates the perceived importance of epistemic concerns and presents opportunities for tools to engage users in dialogue around goals and standards for evidence aggregation.