A case for data valuation transparency via DValCards

TL;DR

This paper highlights the biases and instability of current data valuation methods in ML, demonstrating their ethical and technical issues, and proposes DValCards to promote transparency and responsible use.

Contribution

It introduces DValCards, a framework for transparent data valuation, addressing biases and promoting ethical practices in data markets and ML systems.

Findings

Data pre-processing can significantly alter data value estimates.

Data valuation can increase class imbalance when subsampling.

Underrepresented groups may be undervalued by current metrics.

Abstract

Following the rise in popularity of data-centric machine learning (ML), various data valuation methods have been proposed to quantify the contribution of each datapoint to desired ML model performance metrics (e.g., accuracy). Beyond the technical applications of data valuation methods (e.g., data cleaning, data acquisition, etc.), it has been suggested that within the context of data markets, data buyers might utilize such methods to fairly compensate data owners. Here we demonstrate that data valuation metrics are inherently biased and unstable under simple algorithmic design choices, resulting in both technical and ethical implications. By analyzing 9 tabular classification datasets and 6 data valuation methods, we illustrate how (1) common and inexpensive data pre-processing techniques can drastically alter estimated data values; (2) subsampling via data valuation metrics may increase class imbalance; and (3) data valuation metrics may undervalue underrepresented group data. Consequently, we argue in favor of increased transparency associated with data valuation in-the-wild and introduce the novel Data Valuation Cards (DValCards) framework towards this aim. The proliferation of DValCards will reduce misuse of data valuation metrics, including in data pricing, and build trust in responsible ML systems.