Memorization Control in Diffusion Models from Denoising-centric Perspective

TL;DR

This paper introduces a denoising-centric approach to control memorization in diffusion models by adjusting timestep sampling, which balances memorization and generalization effectively.

Contribution

It proposes a novel timestep sampling strategy that explicitly manages learning focus along the denoising process to reduce memorization.

Findings

Shifting learning emphasis to later denoising steps reduces memorization.

Adjusted sampling improves distributional alignment with training data.

Method is effective on image and 1D signal generation tasks.

Abstract

Controlling memorization in diffusion models is critical for applications that require generated data to closely match the training distribution. Existing approaches mainly focus on data centric or model centric modifications, treating the diffusion model as an isolated predictor. In this paper, we study memorization in diffusion models from a denoising centric perspective. We show that uniform timestep sampling leads to unequal learning contributions across denoising steps due to differences in signal to noise ratio, which biases training toward memorization. To address this, we propose a timestep sampling strategy that explicitly controls where learning occurs along the denoising trajectory. By adjusting the width of the confidence interval, our method provides direct control over the memorization generalization trade off. Experiments on image and 1D signal generation tasks demonstrate that shifting learning emphasis toward later denoising steps consistently reduces memorization and improves distributional alignment with training data, validating the generality and effectiveness of our approach.