Automating Data Science: Prospects and Challenges

TL;DR

This paper discusses the potential and challenges of automating data science tasks, highlighting progress in AutoML and emphasizing the ongoing need for human expertise in complex, context-dependent aspects.

Contribution

It provides an overview of current automation techniques in data science, analyzing their capabilities and limitations, and discusses future prospects and challenges.

Findings

AutoML is transforming modeling stages.

Automation complements rather than replaces data scientists.

Complex tasks still require human judgment.

Abstract

Given the complexity of typical data science projects and the associated demand for human expertise, automation has the potential to transform the data science process. Key insights: * Automation in data science aims to facilitate and transform the work of data scientists, not to replace them. * Important parts of data science are already being automated, especially in the modeling stages, where techniques such as automated machine learning (AutoML) are gaining traction. * Other aspects are harder to automate, not only because of technological challenges, but because open-ended and context-dependent tasks require human interaction.