A Versatile Influence Function for Data Attribution with Non-Decomposable Loss

TL;DR

This paper introduces the Versatile Influence Function (VIF), a new method that extends influence functions to non-decomposable losses in machine learning, enabling efficient data attribution for complex models beyond traditional decomposable loss functions.

Contribution

The paper proposes VIF, a general influence function approach that applies to any non-decomposable loss, leveraging auto-differentiation to avoid case-specific derivations, and demonstrates its effectiveness across diverse applications.

Findings

VIF closely matches brute-force leave-one-out results

VIF is up to 1000 times faster than brute-force methods

VIF works effectively on Cox regression, node embedding, and ranking tasks

Abstract

Influence function, a technique rooted in robust statistics, has been adapted in modern machine learning for a novel application: data attribution -- quantifying how individual training data points affect a model's predictions. However, the common derivation of influence functions in the data attribution literature is limited to loss functions that can be decomposed into a sum of individual data point losses, with the most prominent examples known as M-estimators. This restricts the application of influence functions to more complex learning objectives, which we refer to as non-decomposable losses, such as contrastive or ranking losses, where a unit loss term depends on multiple data points and cannot be decomposed further. In this work, we bridge this gap by revisiting the general formulation of influence function from robust statistics, which extends beyond M-estimators. Based on this formulation, we propose a novel method, the Versatile Influence Function (VIF), that can be straightforwardly applied to machine learning models trained with any non-decomposable loss. In comparison to the classical approach in statistics, the proposed VIF is designed to fully leverage the power of auto-differentiation, hereby eliminating the need for case-specific derivations of each loss function. We demonstrate the effectiveness of VIF across three examples: Cox regression for survival analysis, node embedding for network analysis, and listwise learning-to-rank for information retrieval. In all cases, the influence estimated by VIF closely resembles the results obtained by brute-force leave-one-out retraining, while being up to $10^3$ times faster to compute. We believe VIF represents a significant advancement in data attribution, enabling efficient influence-function-based attribution across a wide range of machine learning paradigms, with broad potential for practical use cases.