Data Acquisition for Improving Machine Learning Models

TL;DR

This paper investigates data acquisition strategies to enhance machine learning model accuracy, formalizing the process and proposing two adaptive algorithms that balance exploration and exploitation, validated through extensive experiments.

Contribution

It introduces two novel data acquisition algorithms that optimize data collection for ML models by balancing exploration and exploitation, supported by comprehensive experimental evaluation.

Findings

Estimation and Allocation effectively estimates data utility and guides data acquisition.

Sequential Predicate Selection adaptively explores data space to improve model accuracy.

Trade-offs between exploration and exploitation are characterized for different scenarios.

Abstract

The vast advances in Machine Learning over the last ten years have been powered by the availability of suitably prepared data for training purposes. The future of ML-enabled enterprise hinges on data. As such, there is already a vibrant market offering data annotation services to tailor sophisticated ML models. In this paper, we present research on the practical problem of obtaining data in order to improve the accuracy of ML models. We consider an environment in which consumers query for data to enhance the accuracy of their models and data providers who possess data make them available for training purposes. We first formalize this interaction process laying out the suitable framework and associated parameters for data exchange. We then propose two data acquisition strategies that consider a trade-off between exploration during which we obtain data to learn about the distribution of a provider's data and exploitation during which we optimize our data inquiries utilizing the gained knowledge. In the first strategy, Estimation and Allocation, we utilize queries to estimate the utilities of various predicates while learning about the distribution of the provider's data; then we proceed to the allocation stage in which we utilize those learned utility estimates to inform our data acquisition decisions. The second algorithmic proposal, named Sequential Predicate Selection, utilizes a sampling strategy to explore the distribution of the provider's data, adaptively investing more resources to parts of the data space that are statistically more promising to improve overall model accuracy. We present a detailed experimental evaluation of our proposals utilizing a variety of ML models and associated real data sets exploring all applicable parameters of interest. We identify trade-offs and highlight the relative benefits of each algorithm to further optimize model accuracy.