Noise Conditional Variational Score Distillation

TL;DR

This paper introduces Noise Conditional Variational Score Distillation (NCVSD), a method that distills diffusion models into efficient generative denoisers capable of fast, high-quality sample generation and flexible inference across noise levels.

Contribution

The paper presents a novel approach that leverages the unconditional score function to distill diffusion models into scalable, versatile denoisers with improved efficiency and sample quality.

Findings

Outperforms larger diffusion models with scaled test-time compute.

Achieves record LPIPS scores on inverse problems with fewer NFEs.

Enables fast one-step and multi-step sampling for high-quality generation.

Abstract

We propose Noise Conditional Variational Score Distillation (NCVSD), a novel method for distilling pretrained diffusion models into generative denoisers. We achieve this by revealing that the unconditional score function implicitly characterizes the score function of denoising posterior distributions. By integrating this insight into the Variational Score Distillation (VSD) framework, we enable scalable learning of generative denoisers capable of approximating samples from the denoising posterior distribution across a wide range of noise levels. The proposed generative denoisers exhibit desirable properties that allow fast generation while preserve the benefit of iterative refinement: (1) fast one-step generation through sampling from pure Gaussian noise at high noise levels; (2) improved sample quality by scaling the test-time compute with multi-step sampling; and (3) zero-shot probabilistic inference for flexible and controllable sampling. We evaluate NCVSD through extensive experiments, including class-conditional image generation and inverse problem solving. By scaling the test-time compute, our method outperforms teacher diffusion models and is on par with consistency models of larger sizes. Additionally, with significantly fewer NFEs than diffusion-based methods, we achieve record-breaking LPIPS on inverse problems.