On Investigating the Conservative Property of Score-Based Generative Models

TL;DR

This paper introduces Quasi-Conservative Score-Based Models (QCSBMs) that combine the strengths of constrained and unconstrained SBMs, addressing their individual limitations to improve generative modeling performance.

Contribution

The paper proposes QCSBMs, a novel framework that maintains the conservativeness property while leveraging flexible architectures, with efficient training via Hutchinson's trace estimator.

Findings

QCSBMs outperform existing SBMs on multiple datasets.

Theoretical analysis confirms the preservation of conservativeness.

Experimental results demonstrate improved generative quality.

Abstract

Existing Score-Based Models (SBMs) can be categorized into constrained SBMs (CSBMs) or unconstrained SBMs (USBMs) according to their parameterization approaches. CSBMs model probability density functions as Boltzmann distributions, and assign their predictions as the negative gradients of some scalar-valued energy functions. On the other hand, USBMs employ flexible architectures capable of directly estimating scores without the need to explicitly model energy functions. In this paper, we demonstrate that the architectural constraints of CSBMs may limit their modeling ability. In addition, we show that USBMs' inability to preserve the property of conservativeness may lead to degraded performance in practice. To address the above issues, we propose Quasi-Conservative Score-Based Models (QCSBMs) for keeping the advantages of both CSBMs and USBMs. Our theoretical derivations demonstrate that the training objective of QCSBMs can be efficiently integrated into the training processes by leveraging the Hutchinson's trace estimator. In addition, our experimental results on the CIFAR-10, CIFAR-100, ImageNet, and SVHN datasets validate the effectiveness of QCSBMs. Finally, we justify the advantage of QCSBMs using an example of a one-layered autoencoder.