Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models

TL;DR

Controlling model capacity via Parameter-Efficient Fine-Tuning effectively reduces memorization in text-conditional diffusion models while improving generation quality, offering a practical mitigation strategy.

Contribution

This work demonstrates that PEFT significantly mitigates memorization in diffusion models compared to full fine-tuning, with added benefits for generation quality.

Findings

PEFT reduces memorization in diffusion models.

PEFT improves downstream generation quality.

PEFT can be combined with other mitigation techniques.

Abstract

In this work, we present compelling evidence that controlling model capacity during fine-tuning can effectively mitigate memorization in diffusion models. Specifically, we demonstrate that adopting Parameter-Efficient Fine-Tuning (PEFT) within the pre-train fine-tune paradigm significantly reduces memorization compared to traditional full fine-tuning approaches. Our experiments utilize the MIMIC dataset, which comprises image-text pairs of chest X-rays and their corresponding reports. The results, evaluated through a range of memorization and generation quality metrics, indicate that PEFT not only diminishes memorization but also enhances downstream generation quality. Additionally, PEFT methods can be seamlessly combined with existing memorization mitigation techniques for further improvement. The code for our experiments is available at: https://github.com/Raman1121/Diffusion_Memorization_HPO