Daunce: Data Attribution through Uncertainty Estimation

TL;DR

Daunce introduces a scalable data attribution method using uncertainty estimation by fine-tuning perturbed models and analyzing loss covariance, outperforming existing methods in accuracy and applicability to large language models.

Contribution

It presents a novel, scalable data attribution technique based on uncertainty estimation, effective for large models and proprietary LLMs, with improved accuracy over prior gradient-based methods.

Findings

Daunce achieves higher attribution accuracy than existing methods.

It is scalable to large language models, including proprietary GPTs.

The method is effective across vision and language tasks.

Abstract

Training data attribution (TDA) methods aim to identify which training examples influence a model's predictions on specific test data most. By quantifying these influences, TDA supports critical applications such as data debugging, curation, and valuation. Gradient-based TDA methods rely on gradients and second-order information, limiting their applicability at scale. While recent random projection-based methods improve scalability, they often suffer from degraded attribution accuracy. Motivated by connections between uncertainty and influence functions, we introduce Daunce - a simple yet effective data attribution approach through uncertainty estimation. Our method operates by fine-tuning a collection of perturbed models and computing the covariance of per-example losses across these models as the attribution score. Daunce is scalable to large language models (LLMs) and achieves more accurate attribution compared to existing TDA methods. We validate Daunce on tasks ranging from vision tasks to LLM fine-tuning, and further demonstrate its compatibility with black-box model access. Applied to OpenAI's GPT models, our method achieves, to our knowledge, the first instance of data attribution on proprietary LLMs.