Debug Smarter, Not Harder: AI Agents for Error Resolution in Computational Notebooks

TL;DR

This paper introduces an AI agent tailored for error resolution in computational notebooks, demonstrating improved user-rated performance over existing solutions, with insights into user interaction challenges and potential for enhancing collaborative data science tools.

Contribution

The paper presents a novel AI agent specifically designed for error fixing in computational notebooks, integrating it into Datalore and evaluating its effectiveness against traditional methods.

Findings

Users rate the agentic system's error resolution higher.

The system interacts with notebooks similarly to a user.

Users face UI challenges with the agent.

Abstract

Computational notebooks became indispensable tools for research-related development, offering unprecedented interactivity and flexibility in the development process. However, these benefits come at the cost of reproducibility and an increased potential for bugs. With the rise of code-fluent Large Language Models empowered with agentic techniques, smart bug-fixing tools with a high level of autonomy have emerged. However, those tools are tuned for classical script programming and still struggle with non-linear computational notebooks. In this paper, we present an AI agent designed specifically for error resolution in a computational notebook. We have developed an agentic system capable of exploring a notebook environment by interacting with it -- similar to how a user would -- and integrated the system into the JetBrains service for collaborative data science called Datalore. We evaluate our approach against the pre-existing single-action solution by comparing costs and conducting a user study. Users rate the error resolution capabilities of the agentic system higher but experience difficulties with UI. We share the results of the study and consider them valuable for further improving user-agent collaboration.