Data Engineering for Everyone

TL;DR

The paper emphasizes the importance of open-source data sets in accelerating machine learning research and discusses the potential of automatic data set generation tools to address data scarcity.

Contribution

It highlights the critical role of open data sets in ML innovation and explores the potential of automation to enhance data set creation.

Findings

Open data sets are widely used in top AI research.

Scarcity of accessible open data sets limits ML progress.

Automatic data set generation could mitigate data scarcity.

Abstract

Data engineering is one of the fastest-growing fields within machine learning (ML). As ML becomes more common, the appetite for data grows more ravenous. But ML requires more data than individual teams of data engineers can readily produce, which presents a severe challenge to ML deployment at scale. Much like the software-engineering revolution, where mass adoption of open-source software replaced the closed, in-house development model for infrastructure code, there is a growing need to enable rapid development and open contribution to massive machine learning data sets. This article shows that open-source data sets are the rocket fuel for research and innovation at even some of the largest AI organizations. Our analysis of nearly 2000 research publications from Facebook, Google and Microsoft over the past five years shows the widespread use and adoption of open data sets. Open data sets that are easily accessible to the public are vital to accelerating ML innovation for everyone. But such open resources are scarce in the wild. So, what if we are able to accelerate data-set creation via automatic data set generation tools?