Shapley Value on Probabilistic Classifiers

TL;DR

This paper introduces P-Shapley, a new data valuation method based on Shapley values that uses class probabilities from probabilistic classifiers, improving the assessment of data importance in ML models.

Contribution

The paper proposes P-Shapley, a novel probabilistic utility function for data valuation that better distinguishes beneficial and detrimental data points in probabilistic classifiers.

Findings

P-Shapley effectively evaluates data importance in probabilistic classifiers.

Calibration functions improve the accuracy of marginal contribution estimates.

Experiments show P-Shapley outperforms traditional methods on real datasets.

Abstract

Data valuation has become an increasingly significant discipline in data science due to the economic value of data. In the context of machine learning (ML), data valuation methods aim to equitably measure the contribution of each data point to the utility of an ML model. One prevalent method is Shapley value, which helps identify data points that are beneficial or detrimental to an ML model. However, traditional Shapley-based data valuation methods may not effectively distinguish between beneficial and detrimental training data points for probabilistic classifiers. In this paper, we propose Probabilistic Shapley (P-Shapley) value by constructing a probability-wise utility function that leverages the predicted class probabilities of probabilistic classifiers rather than binarized prediction results in the traditional Shapley value. We also offer several activation functions for confidence calibration to effectively quantify the marginal contribution of each data point to the probabilistic classifiers. Extensive experiments on four real-world datasets demonstrate the effectiveness of our proposed P-Shapley value in evaluating the importance of data for building a high-usability and trustworthy ML model.