Private, Augmentation-Robust and Task-Agnostic Data Valuation Approach for Data Marketplace

TL;DR

PriArTa is a privacy-preserving, task-agnostic data valuation method that efficiently assesses dataset value in data marketplaces by measuring distribution similarity, robust to data transformations, without exposing full datasets.

Contribution

The paper introduces PriArTa, a novel, communication-efficient, privacy-preserving data valuation approach that is robust to data transformations and applicable across various tasks.

Findings

Effective in real-world image datasets

Maintains privacy while evaluating dataset value

Robust to common data transformations

Abstract

Evaluating datasets in data marketplaces, where the buyer aim to purchase valuable data, is a critical challenge. In this paper, we introduce an innovative task-agnostic data valuation method called PriArTa which is an approach for computing the distance between the distribution of the buyer's existing dataset and the seller's dataset, allowing the buyer to determine how effectively the new data can enhance its dataset. PriArTa is communication-efficient, enabling the buyer to evaluate datasets without needing access to the entire dataset from each seller. Instead, the buyer requests that sellers perform specific preprocessing on their data and then send back the results. Using this information and a scoring metric, the buyer can evaluate the dataset. The preprocessing is designed to allow the buyer to compute the score while preserving the privacy of each seller's dataset, mitigating the risk of information leakage before the purchase. A key feature of PriArTa is its robustness to common data transformations, ensuring consistent value assessment and reducing the risk of purchasing redundant data. The effectiveness of PriArTa is demonstrated through experiments on real-world image datasets, showing its ability to perform privacy-preserving, augmentation-robust data valuation in data marketplaces.