CS-Shapley: Class-wise Shapley Values for Data Valuation in Classification

TL;DR

This paper introduces CS-Shapley, a novel data valuation method for classification that differentiates helpful and harmful instances, improving noisy label detection and data removal tasks, with demonstrated transferability across models.

Contribution

The paper proposes CS-Shapley, a new value function for Shapley-based data valuation that uniquely evaluates in-class and out-of-class contributions in classification tasks.

Findings

CS-Shapley outperforms existing methods in noisy label detection.

The value function is theoretically unique under certain properties.

Data values are transferable across different classifiers.

Abstract

Data valuation, or the valuation of individual datum contributions, has seen growing interest in machine learning due to its demonstrable efficacy for tasks such as noisy label detection. In particular, due to the desirable axiomatic properties, several Shapley value approximation methods have been proposed. In these methods, the value function is typically defined as the predictive accuracy over the entire development set. However, this limits the ability to differentiate between training instances that are helpful or harmful to their own classes. Intuitively, instances that harm their own classes may be noisy or mislabeled and should receive a lower valuation than helpful instances. In this work, we propose CS-Shapley, a Shapley value with a new value function that discriminates between training instances' in-class and out-of-class contributions. Our theoretical analysis shows the proposed value function is (essentially) the unique function that satisfies two desirable properties for evaluating data values in classification. Further, our experiments on two benchmark evaluation tasks (data removal and noisy label detection) and four classifiers demonstrate the effectiveness of CS-Shapley over existing methods. Lastly, we evaluate the "transferability" of data values estimated from one classifier to others, and our results suggest Shapley-based data valuation is transferable for application across different models.