Measuring the Sensitivity of Classification Models with the Error Sensitivity Profile

TL;DR

This paper introduces the Error Sensitivity Profile (ESP), a metric to measure how errors in features affect model performance, aiding targeted data cleaning, supported by a tool suite called extdirty.

Contribution

The paper proposes ESP as a novel metric for quantifying feature-specific error impact and provides an integrated tool suite for its computation.

Findings

Performance degradation is not always predictable from simple correlations.

Extensive experiments on two datasets with 14 models demonstrate ESP's utility.

Data cleaning can be prioritized effectively using ESP.

Abstract

The quality of training data is critical to the performance of machine learning models. In this paper, the Error Sensitivity Profile (ESP) is proposed. It quantifies the sensitivity of model performance to errors in a single feature or in multiple features. By leveraging ESP, data-cleaning efforts can be prioritized based on error types and features most likely to affect model performance. To support the computation of this metric, an integrated suite of tools, called \dirty, is created. We conduct an extensive experimental study on two widely used datasets using 14 classification models, revealing that performance degradation is not always predictable from simple correlations with the target variable.