In-Run Data Shapley for Adam Optimizer

TL;DR

This paper introduces Adam-Aware In-Run Data Shapley, a novel method for data attribution that accurately accounts for the optimizer's influence, especially for adaptive optimizers like Adam, improving fidelity and efficiency.

Contribution

We develop a new Adam-aware data attribution method that approximates true contributions under Adam optimizer, overcoming limitations of SGD-based proxies.

Findings

Achieves near-perfect fidelity to ground-truth contributions ($R > 0.99$).

Retains approximately 95% of standard training throughput.

Outperforms SGD-based baselines in downstream data attribution tasks.

Abstract

Reliable data attribution is essential for mitigating bias and reducing computational waste in modern machine learning, with the Shapley value serving as the theoretical gold standard. While recent "In-Run" methods bypass the prohibitive cost of retraining by estimating contributions dynamically, they heavily rely on the linear structure of Stochastic Gradient Descent (SGD) and fail to capture the complex dynamics of adaptive optimizers like Adam. In this work, we demonstrate that data attribution is inherently optimizer-dependent: we show that SGD-based proxies diverge significantly from true contributions under Adam (Pearson $R \approx 0.11$), rendering them ineffective for modern training pipelines. To bridge this gap, we propose Adam-Aware In-Run Data Shapley. We derive a closed-form approximation that restores additivity by redefining utility under a fixed-state assumption and enable scalable computation via a novel Linearized Ghost Approximation. This technique linearizes the variance-dependent scaling term, allowing us to compute pairwise gradient dot-products without materializing per-sample gradients. Extensive experiments show that our method achieves near-perfect fidelity to ground-truth marginal contributions ($R > 0.99$) while retaining $\sim$95\% of standard training throughput. Furthermore, our Adam-aware attribution significantly outperforms SGD-based baselines in data attribution downstream tasks.