How Much Automation Does a Data Scientist Want?

TL;DR

This study investigates data scientists' automation preferences across the DS/ML lifecycle, revealing varied automation needs and emphasizing user-controlled approaches rather than full automation.

Contribution

It introduces a human-centered AutoML framework with detailed roles, stages, and automation levels, and empirically studies user preferences through a comprehensive survey.

Findings

Different user personas participate in distinct lifecycle stages.

Desired automation levels vary significantly by user role and stage.

Complete end-to-end automation lacks user support.

Abstract

Data science and machine learning (DS/ML) are at the heart of the recent advancements of many Artificial Intelligence (AI) applications. There is an active research thread in AI, \autoai, that aims to develop systems for automating end-to-end the DS/ML Lifecycle. However, do DS and ML workers really want to automate their DS/ML workflow? To answer this question, we first synthesize a human-centered AutoML framework with 6 User Role/Personas, 10 Stages and 43 Sub-Tasks, 5 Levels of Automation, and 5 Types of Explanation, through reviewing research literature and marketing reports. Secondly, we use the framework to guide the design of an online survey study with 217 DS/ML workers who had varying degrees of experience, and different user roles "matching" to our 6 roles/personas. We found that different user personas participated in distinct stages of the lifecycle -- but not all stages. Their desired levels of automation and types of explanation for AutoML also varied significantly depending on the DS/ML stage and the user persona. Based on the survey results, we argue there is no rationale from user needs for complete automation of the end-to-end DS/ML lifecycle. We propose new next steps for user-controlled DS/ML automation.