Filtering Memorization from Parameter-Space in Diffusion Models

TL;DR

This paper introduces BAF, a training-free method to reduce memorization in diffusion model LoRAs by spectral filtering, enhancing privacy without sacrificing output quality.

Contribution

Proposes BAF, a novel post-hoc, data-free spectral filtering technique to mitigate memorization in diffusion LoRAs, applicable without retraining or data access.

Findings

BAF effectively reduces memorization across multiple datasets.

BAF preserves or improves generation quality.

BAF is training-free and data-free, suitable for existing LoRAs.

Abstract

Low-Rank Adaptation (LoRA) has become a widely used mechanism for customizing diffusion models, enabling users to inject new visual concepts or styles through lightweight parameter updates. However, LoRAs can memorize training images, causing generated outputs to reproduce copyrighted or sensitive content. This risk is particularly concerning in LoRA-sharing ecosystems, where users distribute trained LoRAs without releasing the underlying training data. Existing approaches for mitigating memorization rely on access to the training pipeline, training data, or control over the inference process, making them difficult to apply when only the released LoRA weights are available. We propose \textbf{Base-Anchored Filtering (BAF)}, a training-free and data-free framework for post-hoc memorization mitigation in diffusion LoRAs. BAF decomposes LoRA updates into spectral channels and measures their alignment with the principal subspace of the pretrained backbone. Channels strongly aligned with this subspace are retained as generalizable adaptations, while weakly aligned channels are suppressed as potential carriers of memorized content. Experiments on multiple datasets and diffusion backbones demonstrate that BAF consistently reduces memorization while preserving or even improving generation quality. Our code is available in the supplementary material.