MYCROFT: Towards Effective and Efficient External Data Augmentation

TL;DR

Mycroft is a data-efficient method that helps ML practitioners select valuable external data sources with minimal data sharing, improving model performance rapidly and securely across various tasks.

Contribution

It introduces a novel approach combining feature space distances and gradient matching to identify informative data subsets from private sources.

Findings

Converges rapidly to full-data performance

Robust to noise in data sources

Effectively ranks data owners by utility

Abstract

Machine learning (ML) models often require large amounts of data to perform well. When the available data is limited, model trainers may need to acquire more data from external sources. Often, useful data is held by private entities who are hesitant to share their data due to propriety and privacy concerns. This makes it challenging and expensive for model trainers to acquire the data they need to improve model performance. To address this challenge, we propose Mycroft, a data-efficient method that enables model trainers to evaluate the relative utility of different data sources while working with a constrained data-sharing budget. By leveraging feature space distances and gradient matching, Mycroft identifies small but informative data subsets from each owner, allowing model trainers to maximize performance with minimal data exposure. Experimental results across four tasks in two domains show that Mycroft converges rapidly to the performance of the full-information baseline, where all data is shared. Moreover, Mycroft is robust to noise and can effectively rank data owners by utility. Mycroft can pave the way for democratized training of high performance ML models.