FlowBook: Enforcing Reproducibility in Computational Notebooks

TL;DR

FlowBook introduces a dynamic analysis approach to enforce reproducibility in computational notebooks by ensuring top-to-bottom execution from an empty state matches recorded outputs, addressing hidden state issues.

Contribution

It formalizes a reproducibility criterion and implements FlowBook, which tracks read/write sets to detect stale cells and prevent reproducibility violations without complex dependency analysis.

Findings

FlowBook detects stale cells effectively.

Reproducibility enforcement incurs median latency of 70 ms.

The approach avoids tradeoffs of prior dependency analysis methods.

Abstract

Computational notebooks are notoriously prone to reproducibility failures. By permitting out-of-order cell execution, notebooks accumulate hidden state and implicit dependencies that cause interactive executions to silently diverge from clean top-to-bottom runs. Prior approaches either employ dependency analyses or enforce reactive dataflow models that face fundamental tradeoffs among expressiveness, precision, and performance. This paper exploits the insight that reproducibility can be enforced without precise dependency tracking: a notebook is reproducible if and only if executing its cells in top-to-bottom order from an empty store produces exactly the outputs currently recorded. We formalize this notion of reproducibility and present FlowBook, which implements a dynamic analysis that enforces reproducibility by tracking read and write sets at cell boundaries. FlowBook detects stale cells whose recorded outputs may no longer reflect the current notebook state and prevents operations that would violate reproducibility. FlowBook incurs near-imperceptible latency overhead (median: 70 ms).