Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costs

TL;DR

This paper introduces a novel method for estimating covariance in diffusion models that leverages training data and trajectory curvature, eliminating the need for costly test-time computations and improving performance in inverse problems.

Contribution

The authors propose a new framework that uses free covariance information from training data and trajectory curvature, with a transfer method and low-rank updates, to enhance diffusion model guidance.

Findings

Outperforms recent baselines in linear inverse problems

Requires fewer diffusion steps for effective results

Eliminates extra computational costs during testing

Abstract

The covariance for clean data given a noisy observation is an important quantity in many training-free guided generation methods for diffusion models. Current methods require heavy test-time computation, altering the standard diffusion training process or denoiser architecture, or making heavy approximations. We propose a new framework that sidesteps these issues by using covariance information that is available for free from training data and the curvature of the generative trajectory, which is linked to the covariance through the second-order Tweedie's formula. We integrate these sources of information using (i) a novel method to transfer covariance estimates across noise levels and (ii) low-rank updates in a given noise level. We validate the method on linear inverse problems, where it outperforms recent baselines, especially with fewer diffusion steps.