Variational Control for Guidance in Diffusion Models

TL;DR

This paper introduces Diffusion Trajectory Matching (DTM), a variational inference-based guidance method for diffusion models that improves performance on inverse problems without extra training.

Contribution

It presents a unified framework for guidance in diffusion models, enabling new methods and state-of-the-art results on various inverse problems without additional training.

Findings

Achieves state-of-the-art results on inverse problems

Unifies various guidance methods under a single framework

Does not require additional model training

Abstract

Diffusion models exhibit excellent sample quality, but existing guidance methods often require additional model training or are limited to specific tasks. We revisit guidance in diffusion models from the perspective of variational inference and control, introducing Diffusion Trajectory Matching (DTM) that enables guiding pretrained diffusion trajectories to satisfy a terminal cost. DTM unifies a broad class of guidance methods and enables novel instantiations. We introduce a new method within this framework that achieves state-of-the-art results on several linear, non-linear, and blind inverse problems without requiring additional model training or specificity to pixel or latent space diffusion models. Our code will be available at https://github.com/czi-ai/oc-guidance