When Provenance Aids and Complicates Reproducibility Judgments

TL;DR

This paper examines how data and interaction provenance influence reproducibility judgments in visualization, highlighting both their benefits and limitations in conveying meaningful insights.

Contribution

It introduces scenarios demonstrating the complex role of provenance in reproducibility and discusses ways to better relate provenance to visualizations.

Findings

Provenance can both aid and hinder reproducibility assessments.

Interaction and insight provenance provide context but do not guarantee reproducibility.

Provenance's utility depends on how well it captures meaningful aspects of visualization processes.

Abstract

It is well-established that the provenance of a scientific result is important, sometimes more important than the actual result. For computational analyses that involve visualization, this provenance information may contain the steps involved in generating visualizations from raw data. Specifically, data provenance tracks the lineage of data and process provenance tracks the steps executed. In this paper, we argue that the utility of computational provenance may not be as clear-cut as we might like. One common use case for provenance is that the information can be used to reproduce the original result. However, in visualization, the goal is often to communicate results to a user or viewer, and thus the insights obtained are ultimately most important. Viewers can miss important changes or react to unimportant ones. Here, interaction provenance, which tracks a user's actions with a visualization, or insight provenance, which tracks the decision-making process, can help capture what happened but don't remove the issues. In this paper, we present scenarios where provenance impacts reproducibility in different ways. We also explore how provenance and visualizations can be better related.