A Good Score Does not Lead to A Good Generative Model

TL;DR

This paper challenges the assumption that high-quality score functions lead to effective generative models by providing a counter-example where well-learned scores produce only blurred samples, highlighting limitations of SGMs.

Contribution

It presents a counter-example demonstrating that well-learned scores do not necessarily produce high-quality samples, revealing limitations in the theoretical understanding of SGMs.

Findings

SGMs can produce Gaussian-blurred samples despite well-learned scores

Counter-example shows limitations of score-based models in certain settings

SGMs may exhibit memorization and fail to generate diverse samples

Abstract

Score-based Generative Models (SGMs) is one leading method in generative modeling, renowned for their ability to generate high-quality samples from complex, high-dimensional data distributions. The method enjoys empirical success and is supported by rigorous theoretical convergence properties. In particular, it has been shown that SGMs can generate samples from a distribution that is close to the ground-truth if the underlying score function is learned well, suggesting the success of SGM as a generative model. We provide a counter-example in this paper. Through the sample complexity argument, we provide one specific setting where the score function is learned well. Yet, SGMs in this setting can only output samples that are Gaussian blurring of training data points, mimicking the effects of kernel density estimation. The finding resonates a series of recent finding that reveal that SGMs can demonstrate strong memorization effect and fail to generate.