Scalability vs. Utility: Do We Have to Sacrifice One for the Other in Data Importance Quantification?

TL;DR

This paper compares various data importance quantification methods, especially Shapley value approximations, analyzing their utility and scalability across multiple machine learning workflows, and proposes a scalable, effective approach using KNN surrogates.

Contribution

It provides a novel theoretical comparison of importance quantification methods and introduces a scalable Shapley value approximation using KNN surrogates that maintains utility.

Findings

KNN-based Shapley approximation achieves comparable utility to existing methods.

The proposed method offers significant scalability improvements, often by orders of magnitude.

Theoretical analysis justifies the advantage of the KNN surrogate over leave-one-out error.

Abstract

Quantifying the importance of each training point to a learning task is a fundamental problem in machine learning and the estimated importance scores have been leveraged to guide a range of data workflows such as data summarization and domain adaption. One simple idea is to use the leave-one-out error of each training point to indicate its importance. Recent work has also proposed to use the Shapley value, as it defines a unique value distribution scheme that satisfies a set of appealing properties. However, calculating Shapley values is often expensive, which limits its applicability in real-world applications at scale. Multiple heuristics to improve the scalability of calculating Shapley values have been proposed recently, with the potential risk of compromising their utility in real-world applications. \textit{How well do existing data quantification methods perform on existing workflows? How do these methods compare with each other, empirically and theoretically? Must we sacrifice scalability for the utility in these workflows when using these methods?} In this paper, we conduct a novel theoretical analysis comparing the utility of different importance quantification methods, and report extensive experimental studies on existing and proposed workflows such as noisy label detection, watermark removal, data summarization, data acquisition, and domain adaptation. We show that Shapley value approximation based on a $K$NN surrogate over pre-trained feature embeddings obtains comparable utility with existing algorithms while achieving significant scalability improvement, often by orders of magnitude. Our theoretical analysis also justifies its advantage over the leave-one-out error. The code is available at \url{https://github.com/AI-secure/Shapley-Study}.