Why do Machine Learning Notebooks Crash? An Empirical Study on Public Python Jupyter Notebooks

TL;DR

This empirical study analyzes 64,031 public Python ML notebooks to identify common crash types, root causes, and their distribution across ML pipeline stages, revealing ML-specific errors and notebook semantics issues.

Contribution

It provides a comprehensive classification of ML-specific crashes and root causes in notebooks, highlighting the impact of notebook semantics and library misuse on stability.

Findings

Over 40% of crashes are due to API misuse and notebook-specific issues.

Most crashes (over 70%) occur during data preparation, training, and evaluation stages.

TensorFlow, Keras, and Torch are the most error-prone ML libraries.

Abstract

Jupyter notebooks have become central in data science, integrating code, text and output in a flexible environment. With the rise of machine learning (ML), notebooks are increasingly used for prototyping and data analysis. However, due to their dependence on complex ML libraries and the flexible notebook semantics that allow cells to be run in any order, notebooks are susceptible to software bugs that may lead to program crashes. This paper presents a comprehensive empirical study focusing on crashes in publicly available Python ML notebooks. We collect 64,031 notebooks containing 92,542 crashes from GitHub and Kaggle, and manually analyze a sample of 746 crashes across various aspects, including crash types and root causes. Our analysis identifies unique ML-specific crash types, such as tensor shape mismatches and dataset value errors that violate API constraints. Additionally, we highlight unique root causes tied to notebook semantics, including out-of-order execution and residual errors from previous cells, which have been largely overlooked in prior research. Furthermore, we identify the most error-prone ML libraries, and analyze crash distribution across ML pipeline stages. We find that over 40% of crashes stem from API misuse and notebook-specific issues. Crashes frequently occur when using ML libraries like TensorFlow/Keras and Torch. Additionally, over 70% of the crashes occur during data preparation, model training, and evaluation or prediction stages of the ML pipeline, while data visualization errors tend to be unique to ML notebooks.