Rethinking Data Shapley for Data Selection Tasks: Misleads and Merits

TL;DR

This paper critically examines Data Shapley's effectiveness in data selection, revealing its limitations under general utility functions and proposing conditions and heuristics for its reliable application.

Contribution

It introduces a hypothesis testing framework, identifies utility function classes where Data Shapley is optimal, and proposes a heuristic to predict its success in data selection.

Findings

Data Shapley's performance can be no better than random without specific utility constraints.

Within certain utility functions, Data Shapley is proven to be optimal.

A heuristic is proposed to predict Data Shapley's effectiveness in various settings.

Abstract

Data Shapley provides a principled approach to data valuation and plays a crucial role in data-centric machine learning (ML) research. Data selection is considered a standard application of Data Shapley. However, its data selection performance has shown to be inconsistent across settings in the literature. This study aims to deepen our understanding of this phenomenon. We introduce a hypothesis testing framework and show that Data Shapley's performance can be no better than random selection without specific constraints on utility functions. We identify a class of utility functions, monotonically transformed modular functions, within which Data Shapley optimally selects data. Based on this insight, we propose a heuristic for predicting Data Shapley's effectiveness in data selection tasks. Our experiments corroborate these findings, adding new insights into when Data Shapley may or may not succeed.