Think it, Run it: Autonomous ML pipeline generation via self-healing multi-agent AI

TL;DR

This paper presents a multi-agent AI system that automates ML pipeline creation from datasets and natural language goals, enhancing efficiency, robustness, and explainability through integrated self-healing and adaptive learning.

Contribution

It introduces a novel multi-agent architecture combining RAG, explainable recommendation, self-healing, and adaptive learning for end-to-end ML pipeline automation.

Findings

Achieved 84.7% success rate on 150 diverse ML tasks.

Outperformed baseline methods in pipeline success and development time.

Demonstrated robustness improvements via self-healing mechanisms.

Abstract

The purpose of our paper is to develop a unified multi-agent architecture that automates end-to-end machine learning (ML) pipeline generation from datasets and natural-language (NL) goals, improving efficiency, robustness and explainability. A five-agent system is proposed to handle profiling, intent parsing, microservice recommendation, Directed Acyclic Graph (DAG) construction and execution. It integrates code-grounded Retrieval-Augmented Generation (RAG) for microservice understanding, an explainable hybrid recommender combining multiple criteria, a self-healing mechanism using Large Language Model (LLM)-based error interpretation and adaptive learning from execution history. The approach is evaluated on 150 ML tasks across diverse scenarios. The system achieves an 84.7% end-to-end pipeline success rate, outperforming baseline methods. It demonstrates improved robustness through self-healing and reduces workflow development time compared to manual construction. The study introduces a novel integration of code-grounded RAG, explainable recommendation, self-healing execution and adaptive learning within a single architecture, showing that tightly coupled intelligent components can outperform isolated solutions.