Error Identification Strategies for Python Jupyter Notebooks

TL;DR

This study explores how Python Jupyter notebook users identify and understand errors, revealing that despite the unique environment, debugging strategies are similar to traditional programming, with implications for tool design and education.

Contribution

It provides an observational analysis of error detection strategies in Python Jupyter notebooks, extending prior research from R notebooks to Python environments.

Findings

Debugging strategies in Jupyter notebooks are similar to traditional programming environments.

Users employ domain knowledge, statistical understanding, and programming skills to find errors.

Insights can inform improvements in notebook tools and educational practices.

Abstract

Computational notebooks -- such as Jupyter or Colab -- combine text and data analysis code. They have become ubiquitous in the world of data science and exploratory data analysis. Since these notebooks present a different programming paradigm than conventional IDE-driven programming, it is plausible that debugging in computational notebooks might also be different. More specifically, since creating notebooks blends domain knowledge, statistical analysis, and programming, the ways in which notebook users find and fix errors in these different forms might be different. In this paper, we present an exploratory, observational study on how Python Jupyter notebook users find and understand potential errors in notebooks. Through a conceptual replication of study design investigating the error identification strategies of R notebook users, we presented users with Python Jupyter notebooks pre-populated with common notebook errors -- errors rooted in either the statistical data analysis, the knowledge of domain concepts, or in the programming. We then analyzed the strategies our study participants used to find these errors and determined how successful each strategy was at identifying errors. Our findings indicate that while the notebook programming environment is different from the environments used for traditional programming, debugging strategies remain quite similar. It is our hope that the insights presented in this paper will help both notebook tool designers and educators make changes to improve how data scientists discover errors more easily in the notebooks they write.