Fits and Starts: Enterprise Use of AutoML and the Role of Humans in the Loop

TL;DR

This paper explores how organizations use AutoML systems in data science, highlighting the roles of human oversight and visualization, and proposing a framework for different automation levels based on user expertise.

Contribution

It introduces a framework categorizing AutoML usage scenarios and examines the tension between automation speed and human oversight in enterprise contexts.

Findings

Identified three distinct AutoML usage scenarios.

Highlighted the tension between automation speed and human oversight.

Showed that data visualization often poorly balances speed and oversight.

Abstract

AutoML systems can speed up routine data science work and make machine learning available to those without expertise in statistics and computer science. These systems have gained traction in enterprise settings where pools of skilled data workers are limited. In this study, we conduct interviews with 29 individuals from organizations of different sizes to characterize how they currently use, or intend to use, AutoML systems in their data science work. Our investigation also captures how data visualization is used in conjunction with AutoML systems. Our findings identify three usage scenarios for AutoML that resulted in a framework summarizing the level of automation desired by data workers with different levels of expertise. We surfaced the tension between speed and human oversight and found that data visualization can do a poor job balancing the two. Our findings have implications for the design and implementation of human-in-the-loop visual analytics approaches.