Beyond Reproducible Research: Building a Formal Representation of a Data Analysis

TL;DR

This paper proposes creating a formal, logical representation of data analyses to improve understanding, evaluation, and reproducibility beyond traditional code sharing methods.

Contribution

It introduces a formal representation framework for data analysis that captures logical reasoning, enabling analysis evaluation without data and visualization of assumptions.

Findings

Formal representation captures analysis reasoning.

Allows evaluation without data access.

Facilitates understanding of assumptions.

Abstract

Data analyses are often constructed in an imperative manner, where commands representing actions taken on the data are issued sequentially. The publication of these commands, along with the data, is essential to the reproducibility of the analysis by others. However, simply presenting the code and the results of running the code can hide important details about the data analyst's premises, expectations, and assumptions about the data. Understanding this analysis reasoning can be critical to evaluating the quality of an analysis and for suggesting possible improvements. We argue that a formal representation of a data analysis that externalizes its logical construction offers more useful information for statically illustrating an analyst's reasoning. Such a formal representation would allow for the evaluation of some aspects of a data analysis without the need for the data, the visualization of the logical connections leading to a conclusion, and the ability to assess the sensitivity of an analyst's assumptions to unexpected features in the data. In this paper we describe an implementation of this formal representation and how it might be applied to some common data analysis tasks.