Influence Functions for Scalable Data Attribution in Diffusion Models

TL;DR

This paper introduces a scalable influence functions framework for data attribution in diffusion models, enabling better interpretability and understanding of how training data affects generated outputs.

Contribution

It develops a novel influence functions approach tailored for diffusion models, including scalable Hessian approximations, and unifies previous methods within this framework.

Findings

Our method outperforms previous data attribution techniques.

Scalable Hessian approximations enable efficient influence computations.

The framework improves interpretability of diffusion models.

Abstract

Diffusion models have led to significant advancements in generative modelling. Yet their widespread adoption poses challenges regarding data attribution and interpretability. In this paper, we aim to help address such challenges in diffusion models by developing an influence functions framework. Influence function-based data attribution methods approximate how a model's output would have changed if some training data were removed. In supervised learning, this is usually used for predicting how the loss on a particular example would change. For diffusion models, we focus on predicting the change in the probability of generating a particular example via several proxy measurements. We show how to formulate influence functions for such quantities and how previously proposed methods can be interpreted as particular design choices in our framework. To ensure scalability of the Hessian computations in influence functions, we systematically develop K-FAC approximations based on generalised Gauss-Newton matrices specifically tailored to diffusion models. We recast previously proposed methods as specific design choices in our framework and show that our recommended method outperforms previous data attribution approaches on common evaluations, such as the Linear Data-modelling Score (LDS) or retraining without top influences, without the need for method-specific hyperparameter tuning.