Guess & Guide: Gradient-Free Zero-Shot Diffusion Guidance

TL;DR

This paper introduces a gradient-free approach for guiding pretrained diffusion models in Bayesian inverse problems, significantly reducing computational costs while maintaining high performance across diverse tasks.

Contribution

We propose a lightweight likelihood surrogate that removes the need for gradient calculations, enabling faster and more efficient zero-shot diffusion guidance.

Findings

Inference cost drops dramatically with our method

Achieves the highest results in multiple inverse problem tasks

Provides the fastest and Pareto optimal guidance approach

Abstract

Pretrained diffusion models serve as effective priors for Bayesian inverse problems. These priors enable zero-shot generation by sampling from the conditional distribution, which avoids the need for task-specific retraining. However, a major limitation of existing methods is their reliance on surrogate likelihoods that require vector-Jacobian products at each denoising step, creating a substantial computational burden. To address this, we introduce a lightweight likelihood surrogate that eliminates the need to calculate gradients through the denoiser network. This enables us to handle diverse inverse problems without backpropagation overhead. Experiments confirm that using our method, the inference cost drops dramatically. At the same time, our approach delivers the highest results in multiple tasks. Broadly speaking, we propose the fastest and Pareto optimal method for Bayesian inverse problems.