The Value of Covariance Matching in Gaussian DDPMs and the Lanczos Sampler

TL;DR

This paper demonstrates that matching the full posterior covariance in Gaussian DDPMs significantly reduces path-space KL divergence, and introduces the Lanczos Gaussian sampler for efficient, matrix-free covariance sampling, improving sample quality.

Contribution

It shows full covariance matching reduces path-KL error from O(1/T) to O(1/T^2) and introduces the Lanczos Gaussian sampler for practical, efficient covariance sampling without dense matrices.

Findings

Matching full covariance reduces path-KL error from O(1/T) to O(1/T^2).

Lanczos Gaussian sampler achieves exponential decay in approximation error with few steps.

Empirical results show improved sample quality with only three Lanczos steps.

Abstract

A central error measure in Gaussian DDPMs is the path-space KL divergence between the exact reverse chain and the learned Gaussian reverse process. This quantity is especially relevant for procedures such as classifier guidance, which perturb the entire reverse trajectory rather than only the terminal sample. Prior analyses show that standard isotropic reverse covariances suffer an unavoidable $\Omega(1/T)$ path-KL error as the number of denoising steps $T$ grows. We show that matching the full posterior covariance breaks this barrier, yielding an order-wise improvement that reduces the path KL to $O(1/T^2)$. To make full covariance matching practical, we introduce the Lanczos Gaussian sampler (LGS), a training-free, matrix-free method for sampling from the optimal reverse covariance using only covariance-vector products, which are available through Jacobian-vector products of the posterior mean. LGS avoids dense covariance storage and auxiliary covariance models. We prove that LGS approximation error decays exponentially in the number of Lanczos steps, where each Lanczos step requires a single Jacobian-vector product. Empirically, using only just three such steps improves sample quality over strong diagonal-covariance baselines, including OCM-DDPM, across standard image benchmarks. This identifies full covariance matching as both theoretically valuable and practically accessible for fast DDPM sampling.