Rescaled Influence Functions: Accurate Data Attribution in High Dimension

TL;DR

This paper introduces rescaled influence functions (RIF), an improved method for data attribution in high-dimensional models that significantly outperforms traditional influence functions in accuracy and robustness against data poisoning.

Contribution

The paper proposes RIF as a drop-in replacement for influence functions, providing more accurate data attribution in high-dimensional settings with minimal computational cost.

Findings

RIF outperforms IF in real-world data attribution tasks.

RIF provides more accurate predictions of data influence.

RIF detects data poisoning attacks that fool IF.

Abstract

How does the training data affect a model's behavior? This is the question we seek to answer with data attribution. The leading practical approaches to data attribution are based on influence functions (IF). IFs utilize a first-order Taylor approximation to efficiently predict the effect of removing a set of samples from the training set without retraining the model, and are used in a wide variety of machine learning applications. However, especially in the high-dimensional regime (# params $\geq \Omega($# samples$)$), they are often imprecise and tend to underestimate the effect of sample removals, even for simple models such as logistic regression. We present rescaled influence functions (RIF), a new tool for data attribution which can be used as a drop-in replacement for influence functions, with little computational overhead but significant improvement in accuracy. We compare IF and RIF on a range of real-world datasets, showing that RIFs offer significantly better predictions in practice, and present a theoretical analysis explaining this improvement. Finally, we present a simple class of data poisoning attacks that would fool IF-based detections but would be detected by RIF.