Data Attribution for Text-to-Image Models by Unlearning Synthesized Images

TL;DR

This paper introduces an efficient method for data attribution in text-to-image models by simulating unlearning of synthesized images, enabling identification of influential training images without costly retraining.

Contribution

The authors propose a novel unlearning simulation technique that accurately identifies influential training images in text-to-image models without extensive retraining.

Findings

The method accurately identifies influential images compared to retraining from scratch.

It reduces computational costs significantly over previous influence estimation methods.

The approach maintains unrelated concepts during unlearning simulation.

Abstract

The goal of data attribution for text-to-image models is to identify the training images that most influence the generation of a new image. Influence is defined such that, for a given output, if a model is retrained from scratch without the most influential images, the model would fail to reproduce the same output. Unfortunately, directly searching for these influential images is computationally infeasible, since it would require repeatedly retraining models from scratch. In our work, we propose an efficient data attribution method by simulating unlearning the synthesized image. We achieve this by increasing the training loss on the output image, without catastrophic forgetting of other, unrelated concepts. We then identify training images with significant loss deviations after the unlearning process and label these as influential. We evaluate our method with a computationally intensive but "gold-standard" retraining from scratch and demonstrate our method's advantages over previous methods.