On the Inflation of KNN-Shapley Value

TL;DR

This paper introduces Calibrated KNN-Shapley, a method to address value inflation in data valuation by calibrating the threshold for beneficial samples, improving detection of detrimental data across various learning scenarios.

Contribution

It proposes CKNN-Shapley, a calibration technique that mitigates value inflation in KNN-Shapley, enhancing data quality assessment in multiple practical learning settings.

Findings

CKNN-Shapley effectively reduces data valuation inflation.

It improves detection of detrimental samples in diverse scenarios.

The method extends to learning with noisy, streaming, and actively labeled data.

Abstract

Shapley value-based data valuation methods, originating from cooperative game theory, quantify the usefulness of each individual sample by considering its contribution to all possible training subsets. Despite their extensive applications, these methods encounter the challenge of value inflation - while samples with negative Shapley values are detrimental, some with positive values can also be harmful. This challenge prompts two fundamental questions: the suitability of zero as a threshold for distinguishing detrimental from beneficial samples and the determination of an appropriate threshold. To address these questions, we focus on KNN-Shapley and propose Calibrated KNN-Shapley (CKNN-Shapley), which calibrates zero as the threshold to distinguish detrimental samples from beneficial ones by mitigating the negative effects of small-sized training subsets. Through extensive experiments, we demonstrate the effectiveness of CKNN-Shapley in alleviating data valuation inflation, detecting detrimental samples, and assessing data quality. We also extend our approach beyond conventional classification settings, applying it to diverse and practical scenarios such as learning with mislabeled data, online learning with stream data, and active learning for label annotation.