Implicit Regularisation in Diffusion Models: An Algorithm-Dependent Generalisation Analysis

TL;DR

This paper develops an algorithm-dependent generalisation theory for diffusion models, introducing score stability to explain how various implicit regularisation techniques improve model generalisation in high-dimensional settings.

Contribution

It introduces the concept of score stability and provides a framework for understanding implicit regularisation in diffusion models beyond model structure.

Findings

Score stability bounds relate to generalisation performance.

Early stopping and discretisation act as implicit regularisers.

SGD optimisation contributes to model regularisation.

Abstract

The success of denoising diffusion models raises important questions regarding their generalisation behaviour, particularly in high-dimensional settings. Notably, it has been shown that when training and sampling are performed perfectly, these models memorise training data -- implying that some form of regularisation is essential for generalisation. Existing theoretical analyses primarily rely on algorithm-independent techniques such as uniform convergence, heavily utilising model structure to obtain generalisation bounds. In this work, we instead leverage the algorithmic aspects that promote generalisation in diffusion models, developing a general theory of algorithm-dependent generalisation for this setting. Borrowing from the framework of algorithmic stability, we introduce the notion of score stability, which quantifies the sensitivity of score-matching algorithms to dataset perturbations. We derive generalisation bounds in terms of score stability, and apply our framework to several fundamental learning settings, identifying sources of regularisation. In particular, we consider denoising score matching with early stopping (denoising regularisation), sampler-wide coarse discretisation (sampler regularisation) and optimising with SGD (optimisation regularisation). By grounding our analysis in algorithmic properties rather than model structure, we identify multiple sources of implicit regularisation unique to diffusion models that have so far been overlooked in the literature.