A Framework to capture and reproduce the Absolute State of Jupyter Notebooks

TL;DR

This paper introduces a framework using Jupyter's extension mechanisms to capture and reproduce the complete, executable state of notebooks, enhancing reproducibility across different environments with minimal performance impact.

Contribution

It presents a novel system for archiving and restoring the full state of Jupyter Notebooks, including code, environment, and runtime variables, using standard extensions.

Findings

Minimal execution overhead observed

Successful replication of notebook state across environments

Enhanced reproducibility of computational research

Abstract

Jupyter Notebooks are an enormously popular tool for creating and narrating computational research projects. They also have enormous potential for creating reproducible scientific research artifacts. Capturing the complete state of a notebook has additional benefits; for instance, the notebook execution may be split between local and remote resources, where the latter may have more powerful processing capabilities or store large or access-limited data. There are several challenges for making notebooks fully reproducible when examined in detail. The notebook code must be replicated entirely, and the underlying Python runtime environments must be identical. More subtle problems arise in replicating referenced data, external library dependencies, and runtime variable states. This paper presents solutions to these problems using Juptyer's standard extension mechanisms to create an archivable system state for a running notebook. We show that the overhead for these additional mechanisms, which involve interacting with the underlying Linux kernel, does not introduce substantial execution time overheads, demonstrating the approach's feasibility.