Automatic Feasibility Study via Data Quality Analysis for ML: A Case-Study on Label Noise

TL;DR

This paper introduces Snoopy, a data quality analysis tool that estimates the Bayes error rate to assess task feasibility in ML projects, helping data scientists avoid unrealistic expectations caused by noisy data.

Contribution

The paper presents a practical Bayes error estimator and demonstrates its effectiveness in feasibility studies, reducing labeling efforts and improving project planning in ML.

Findings

The estimator accurately predicts irreducible error across diverse datasets.

Incorporating feasibility analysis reduces labeling time and costs.

Systematic feasibility studies improve ML project success rates.

Abstract

In our experience of working with domain experts who are using today's AutoML systems, a common problem we encountered is what we call "unrealistic expectations" -- when users are facing a very challenging task with a noisy data acquisition process, while being expected to achieve startlingly high accuracy with machine learning (ML). Many of these are predestined to fail from the beginning. In traditional software engineering, this problem is addressed via a feasibility study, an indispensable step before developing any software system. In this paper, we present Snoopy, with the goal of supporting data scientists and machine learning engineers performing a systematic and theoretically founded feasibility study before building ML applications. We approach this problem by estimating the irreducible error of the underlying task, also known as the Bayes error rate (BER), which stems from data quality issues in datasets used to train or evaluate ML model artifacts. We design a practical Bayes error estimator that is compared against baseline feasibility study candidates on 6 datasets (with additional real and synthetic noise of different levels) in computer vision and natural language processing. Furthermore, by including our systematic feasibility study with additional signals into the iterative label cleaning process, we demonstrate in end-to-end experiments how users are able to save substantial labeling time and monetary efforts.