Evaluation of Large Language Model-Driven AutoML in Data and Model Management from Human-Centered Perspective

TL;DR

This study demonstrates that large language model-driven AutoML frameworks significantly enhance ML implementation success, reduce development time, and improve accuracy across diverse organizational roles through natural language interfaces.

Contribution

It provides empirical evidence that LLM-based AutoML improves accessibility, efficiency, and effectiveness of ML deployment in organizations, bridging technical skill gaps.

Findings

93.34% of users achieved superior performance with LLM-based AutoML

Implementation time reduced by 50% across all expertise levels

Error resolution time decreased by 73% with LLM interfaces

Abstract

As organizations increasingly seek to leverage machine learning (ML) capabilities, the technical complexity of implementing ML solutions creates significant barriers to adoption and impacts operational efficiency. This research examines how Large Language Models (LLMs) can transform the accessibility of ML technologies within organizations through a human-centered Automated Machine Learning (AutoML) approach. Through a comprehensive user study involving 15 professionals across various roles and technical backgrounds, we evaluate the organizational impact of an LLM-based AutoML framework compared to traditional implementation methods. Our research offers four significant contributions to both management practice and technical innovation: First, we present pioneering evidence that LLM-based interfaces can dramatically improve ML implementation success rates, with 93.34% of users achieved superior performance in the LLM condition, with 46.67% showing higher accuracy (10-25% improvement over baseline) and 46.67% demonstrating significantly higher accuracy (>25% improvement over baseline), while 6.67% maintained comparable performance levels; and 60% reporting substantially reduced development time. Second, we demonstrate how natural language interfaces can effectively bridge the technical skills gap in organizations, cutting implementation time by 50% while improving accuracy across all expertise levels. Third, we provide valuable insights for organizations designing human-AI collaborative systems, showing that our approach reduced error resolution time by 73% and significantly accelerated employee learning curves. Finally, we establish empirical support for natural language as an effective interface for complex technical systems, offering organizations a path to democratize ML capabilities without compromising quality or performance.