Understanding Feedback Mechanisms in Machine Learning Jupyter Notebooks

TL;DR

This paper investigates feedback mechanisms in machine learning development using Jupyter notebooks, revealing prevalent implicit feedback and proposing automated validation to improve workflow reliability and reproducibility.

Contribution

It is the first to systematically analyze feedback mechanisms in ML notebooks, categorizing implicit and explicit types and highlighting opportunities for automation and better documentation.

Findings

Implicit feedback dominates critical design decisions.

Explicit feedback mechanisms are underused.

Automated validation via assertions can improve ML workflow reliability.

Abstract

The machine learning development lifecycle is characterized by iterative and exploratory processes that rely on feedback mechanisms to ensure data and model integrity. Despite the critical role of feedback in machine learning engineering, no prior research has been conducted to identify and understand these mechanisms. To address this knowledge gap, we mine 297.8 thousand Jupyter notebooks and analyse 2.3 million code cells. We identify three key feedback mechanisms -- assertions, print statements and last cell statements -- and further categorize them into implicit and explicit forms of feedback. Our findings reveal extensive use of implicit feedback for critical design decisions and the relatively limited adoption of explicit feedback mechanisms. By conducting detailed case studies with selected feedback instances, we uncover the potential for automated validation of critical assumptions in ML workflows using assertions. Finally, this study underscores the need for improved documentation, and provides practical recommendations on how existing feedback mechanisms in the ML development workflow can be effectively used to mitigate technical debt and enhance reproducibility.