Revisiting Data Attribution for Influence Functions

TL;DR

This paper reviews the use of influence functions for data attribution in deep learning, discussing their theoretical basis, recent algorithmic improvements, and effectiveness in identifying influential training data and mislabels.

Contribution

It provides a comprehensive overview of influence functions in deep learning, including recent advances and challenges, to enhance data attribution and model interpretability.

Findings

Influence functions can effectively identify influential training samples.

Recent algorithms improve the efficiency of influence function computations.

Influence functions show promise for mislabel detection in large-scale models.

Abstract

The goal of data attribution is to trace the model's predictions through the learning algorithm and back to its training data. thereby identifying the most influential training samples and understanding how the model's behavior leads to particular predictions. Understanding how individual training examples influence a model's predictions is fundamental for machine learning interpretability, data debugging, and model accountability. Influence functions, originating from robust statistics, offer an efficient, first-order approximation to estimate the impact of marginally upweighting or removing a data point on a model's learned parameters and its subsequent predictions, without the need for expensive retraining. This paper comprehensively reviews the data attribution capability of influence functions in deep learning. We discuss their theoretical foundations, recent algorithmic advances for efficient inverse-Hessian-vector product estimation, and evaluate their effectiveness for data attribution and mislabel detection. Finally, highlighting current challenges and promising directions for unleashing the huge potential of influence functions in large-scale, real-world deep learning scenarios.