DeepFix: Debugging and Fixing Machine Learning Workflow using Agentic AI

TL;DR

DeepFix is an AI-powered tool that automates testing of machine learning pipelines, identifies bugs, explains failures, and suggests fixes, thereby improving reliability and interpretability of ML systems.

Contribution

This paper introduces DeepFix, a novel agentic AI framework for automated testing, bug reporting, and fixing guidance in machine learning workflows.

Findings

DeepFix successfully detects hidden failure modes in ML pipelines.

The tool provides interpretable bug explanations and potential fixes.

Validated on multiple ML systems with promising results.

Abstract

In recent years, machine learning (ML) based software systems are increasingly deployed in several critical applications, yet systematic testing of their behavior remains challenging due to complex model architectures, large input spaces, and evolving deployment environments. Existing testing approaches often rely on generating test cases based on given requirements, which often fail to reveal critical bugs of modern ML models due to their complex nature. Most importantly, such approaches, although they can be used to detect the presence of specific failures in the ML software, they hardly provide any message as to how to fix such errors. To tackle this, in this paper, we present DeepFix, a tool for automated testing of the entire ML pipeline using an agentic AI framework. Our testing approach first leverages Deepchecks to test the ML software for any potential bugs, and thereafter, uses an agentic AI-based approach to generate a detailed bug report. This includes a ranking, based on the severity of the found bugs, along with their explanations, which can be interpreted easily by any non-data science experts and most importantly, also provides possible ways to fix these bugs. Additionally, DeepFix supports several types of ML software systems and can be integrated easily to any ML workflow, enabling continuous testing throughout the development lifecycle. We discuss our already validated cases as well as some planned validations designed to demonstrate how the agentic testing process can reveal hidden failure modes that remain undetected by conventional testing methods. A 5-minute screencast demonstrating the tool's core functionality is available at https://youtu.be/WfwZmFcQgBQ.