Truthful Data Acquisition via Peer Prediction

TL;DR

This paper introduces mechanisms for purchasing data from providers that incentivize truthful reporting using peer prediction techniques, ensuring accuracy, individual rationality, and budget feasibility in different data collection settings.

Contribution

It develops novel peer prediction-based mechanisms for data acquisition that guarantee truthful reporting and accuracy without test data, applicable in single and repeated collection scenarios.

Findings

Mechanisms guarantee truthful reporting as an equilibrium.

They discourage misreports that impair prediction accuracy.

Guarantee individual rationality and budget feasibility under certain conditions.

Abstract

We consider the problem of purchasing data for machine learning or statistical estimation. The data analyst has a budget to purchase datasets from multiple data providers. She does not have any test data that can be used to evaluate the collected data and can assign payments to data providers solely based on the collected datasets. We consider the problem in the standard Bayesian paradigm and in two settings: (1) data are only collected once; (2) data are collected repeatedly and each day's data are drawn independently from the same distribution. For both settings, our mechanisms guarantee that truthfully reporting one's dataset is always an equilibrium by adopting techniques from peer prediction: pay each provider the mutual information between his reported data and other providers' reported data. Depending on the data distribution, the mechanisms can also discourage misreports that would lead to inaccurate predictions. Our mechanisms also guarantee individual rationality and budget feasibility for certain underlying distributions in the first setting and for all distributions in the second setting.