On the Interpolation Effect of Score Smoothing in Diffusion Models

TL;DR

This paper investigates how score smoothing in diffusion models leads to data interpolation, revealing the underlying mechanisms and effects on data generation, supported by analytical and empirical evidence.

Contribution

It provides a theoretical and empirical analysis of score smoothing effects in diffusion models, especially in one-dimensional and simple nonlinear manifold settings.

Findings

Score smoothing causes denoised samples to interpolate training data.

Neural network learning of score functions naturally achieves interpolation effects.

Theoretical analysis explains the interplay between score smoothing and denoising dynamics.

Abstract

Diffusion models have achieved remarkable progress in various domains with an intriguing ability to produce new data that do not exist in the training set. In this work, we study the hypothesis that such creativity arises from the neural network backbone learning a smoothed version of the empirical score function, which guides the denoising dynamics to generate data points that interpolate the training data. Focusing mainly on settings where the training set lies uniformly in a one-dimensional subspace, we elucidate the interplay between score smoothing and the denoising dynamics with analytical solutions and numerical experiments, demonstrating how smoothing the score function can cause the denoised data samples to interpolate the training set along the subspace. Moreover, we present theoretical and empirical evidence that learning score functions with neural networks - either with or without explicit regularization - can naturally achieve a similar effect, including when the data belong to simple nonlinear manifolds.