When Are Reactive Notebooks Not Reactive?

TL;DR

This paper introduces Rex, a test suite to evaluate reactivity in notebook systems, revealing inconsistencies and failures that hinder user understanding and system reliability.

Contribution

It provides a systematic way to assess reactivity in notebooks, highlighting issues and guiding improvements in reactive notebook implementations.

Findings

Rex identifies various failure modes in existing reactive notebooks.

Different systems define reactivity in inconsistent ways.

Reactivity failures can break user mental models.

Abstract

Computational notebooks are convenient for programmers, but can easily become confusing and inconsistent due to the ability to incrementally edit a program that is running. Recent reactive notebook systems, such as Ipyflow, Marimo and Observable, strive to keep notebook state in sync with the current cell code by re-executing a minimal set of cells upon modification. However, each system defines reactivity a different way. Additionally, within any definition, we find simple notebook modifications that can break each system. Overall, these inconsistencies make it difficult for users to construct a mental model of their reactive notebook's implementation. This paper proposes Rex, a fine-grained test suite to discuss and assess reactivity capabilities within reactive notebook systems. We evaluate Rex on three existing reactive notebook systems and classify their failures with the aims of (i) helping programmers understand when reactivity fails and (ii) helping notebook implementations improve.