Data Shapley in One Training Run

TL;DR

This paper introduces In-Run Data Shapley, a scalable method for attributing data contribution to a specific model obtained from a single training run, enabling efficient analysis of large-scale models and pretraining data.

Contribution

It proposes a novel, efficient approach to data attribution that works within a single training run, overcoming the computational limitations of existing Data Shapley methods.

Findings

Enables data attribution for large models during pretraining.

Achieves negligible additional runtime compared to standard training.

Provides insights into pretraining data's contribution and copyright implications.

Abstract

Data Shapley provides a principled framework for attributing data's contribution within machine learning contexts. However, existing approaches require re-training models on different data subsets, which is computationally intensive, foreclosing their application to large-scale models. Furthermore, they produce the same attribution score for any models produced by running the learning algorithm, meaning they cannot perform targeted attribution towards a specific model obtained from a single run of the algorithm. This paper introduces In-Run Data Shapley, which addresses these limitations by offering scalable data attribution for a target model of interest. In its most efficient implementation, our technique incurs negligible additional runtime compared to standard model training. This dramatic efficiency improvement makes it possible to perform data attribution for the foundation model pretraining stage for the first time. We present several case studies that offer fresh insights into pretraining data's contribution and discuss their implications for copyright in generative AI and pretraining data curation.