Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining

TL;DR

This paper investigates the hyperparameter sensitivity of data attribution methods, highlighting the challenge of tuning these parameters efficiently without costly retraining, and proposes a theoretical approach for practical hyperparameter selection.

Contribution

It provides the first large-scale empirical study on hyperparameter sensitivity in data attribution and introduces a theoretical method for hyperparameter tuning without retraining.

Findings

Most data attribution methods are sensitive to key hyperparameters.

Evaluating attribution performance is costly due to retraining requirements.

A lightweight, theory-based hyperparameter selection procedure is effective across benchmarks.

Abstract

Data attribution methods, which quantify the influence of individual training data points on a machine learning model, have gained increasing popularity in data-centric applications in modern AI. Despite a recent surge of new methods developed in this space, the impact of hyperparameter tuning in these methods remains under-explored. In this work, we present the first large-scale empirical study to understand the hyperparameter sensitivity of common data attribution methods. Our results show that most methods are indeed sensitive to certain key hyperparameters. However, unlike typical machine learning algorithms -- whose hyperparameters can be tuned using computationally-cheap validation metrics -- evaluating data attribution performance often requires retraining models on subsets of training data, making such metrics prohibitively costly for hyperparameter tuning. This poses a critical open challenge for the practical application of data attribution methods. To address this challenge, we advocate for better theoretical understandings of hyperparameter behavior to inform efficient tuning strategies. As a case study, we provide a theoretical analysis of the regularization term that is critical in many variants of influence function methods. Building on this analysis, we propose a lightweight procedure for selecting the regularization value without model retraining, and validate its effectiveness across a range of standard data attribution benchmarks. Overall, our study identifies a fundamental yet overlooked challenge in the practical application of data attribution, and highlights the importance of careful discussion on hyperparameter selection in future method development.