Is Data Shapley Not Better than Random in Data Selection? Ask NASH

TL;DR

This paper introduces NASH, a framework that improves data selection by decomposing utility functions into Shapley-informative parts and aggregating them non-linearly, enhancing effectiveness over traditional Data Shapley methods.

Contribution

The paper proposes NASH, a novel method that decomposes utility functions into informative components and combines them non-linearly to improve data subset selection.

Findings

NASH significantly outperforms traditional Data Shapley in data selection tasks.

NASH achieves these improvements with minimal additional computational cost.

The framework effectively identifies high-quality data subsets in various settings.

Abstract

Data selection studies the problem of identifying high-quality subsets of training data. While some existing works have considered selecting the subset of data with top-$m$ Data Shapley or other semivalues as they account for the interaction among every subset of data, other works argue that Data Shapley can sometimes perform ineffectively in practice and select subsets that are no better than random. This raises the questions: (I) Are there certain "Shapley-informative" settings where Data Shapley consistently works well? (II) Can we strategically utilize these settings to select high-quality subsets consistently and efficiently? In this paper, we propose a novel data selection framework, NASH (Non-linear Aggregation of SHapley-informative components), which (I) decomposes the target utility function (e.g., validation accuracy) into simpler, Shapley-informative component functions, and selects data by optimizing an objective that (II) aggregates these components non-linearly. We demonstrate that NASH substantially boosts the effectiveness of Shapley/semivalue-based data selection with minimal additional runtime cost.